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Abstract 

Modernization  is  a  critical  component  of  the  current  transfonnation  effort  within 
the  Department  of  Defense  (DoD).  Effective  and  efficient  modernization  planning  will 
provide  for  the  improved  allocation  of  limited  funding.  The  Air  Force  currently  conducts 
capabilities  based  modernization  planning  to  identify  shortfalls.  Air  Combat  Command 
(ACC)  utilizes  multi-objective  decision  analysis  (MOD A)  techniques  to  support  the 
modernization  planning  process  (MPP).  A  MOD  A  model  has  been  created  to  identify 
and  quantify  capability  shortfalls  across  a  diverse  range  of  mission  areas.  Groups  of 
subject  matter  experts  are  utilized  to  provide  model  inputs  improving  the  usefulness  and 
credibility  of  the  model. 

The  intent  of  this  research  effort  is  to  document  the  ACC  modernization  model 
and  provide  insight  into  their  use  of  groups.  A  methodology  is  created  to  identify 
appropriate  group  decision  making  techniques  for  use  in  MODA.  The  resulting 
taxonomy  table  is  then  used  to  analyze  the  group  decision  process  used  for  the  ACC 
model.  The  documentation  of  the  model  provides  a  reference  of  MODA  use  in 
modernization  planning.  The  methodology  created  will  provide  a  reference  for  the  use  of 
group  decision  making  techniques  in  MODA.  The  identification  of  areas  where  group 
decision  making  techniques  can  be  applied  to  the  ACC  model  provides  insight  capable  of 
strengthening  the  model  and  its  output.  This  will  provide  improved  quantitative 
information  to  modernization  decision  makers. 
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COMBAT  AIR  FORCES  CAMPAIGN  LEVEL  MODERNIZATION  PLANNING: 


A  STUDY  IN  GROUP  DECISION  MAKING 


I.  Introduction 


Overview 

The  issue  of  modernization  is  of  critical  importance  to  the  United  States  Air 
Force.  The  Air  Force  defines  modernization  as  “planned  increases  in  technical 
sophistication  of  forces,  units,  weapon  systems,  and  equipment”  (AFPD  90-11,  2000:1 1). 
The  aging  of  legacy  weapons  systems  and  the  high  demands  placed  on  the  force  due  to 
the  war  on  terrorism  accentuate  the  Air  Force’s  need  to  continue  improving  its 
capabilities.  Increases  in  technology  are  essential  in  ensuring  the  Air  Force  can  continue 
to  dominate  as  the  global  leader  in  aerospace  power. 

Modernization  of  the  Air  Force  is  one  critical  part  of  the  Department  of  Defense 
(DoD)  effort  to  transform  the  military  forces  of  the  United  States.  “In  Air  Force 
parlance,  transformation  means  a  fundamental  change  that  yields  “order-of-magnitude” 
leaps  in  power  rather  than  incremental  gains”  (Dudney,  2002).  Transformation  is  by  no 
means  a  new  concept,  but  has  become  critical  due  to  the  changing  threat  environment  and 
the  aging  of  military  equipment. 

The  transformation  goal  of  the  Air  Force  is  to  be  a  “capabilities-focused 
expeditionary  air  and  space  force”  (Himes,  2002).  In  order  to  meet  this  goal,  the 
research,  development,  and  procurement  of  weapon  systems  is  a  focus  of  Air  Force 
transformation  efforts  (Dudney,  2002).  Transformation  is  accomplished  in  part  through 
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modernization  to  improve  capabilities.  This  will  allow  the  Air  Force  to  maintain  the 
advantage  over  all  current  and  future  enemies. 

The  task  of  modernizing  the  Air  Force  is  formidable  due  to  the  size  of  the  force, 
the  high  level  of  technology  involved,  and  the  complexity  of  the  operating  environment. 
The  Modernization  Planning  Process  (MPP)  requires  participation  at  all  levels  within  the 
Air  Force.  The  MPP  is  based  primarily  on  the  Air  Force  Strategic  Plan  (AFSP)  and 
incorporates  inputs  from  the  actual  war  lighters  through  a  requirements  generation 
process.  The  output  of  the  MPP  is  of  critical  importance  to  the  decision  making  process 
faced  by  Air  Force  leaders. 

“Modernization  plans  identify  current  and  future  capabilities,  deficiencies  in  those 
capabilities,  and  recommended  solutions  to  noted  shortfalls”  (AFPD  90-11,  2000:3).  Air 
Force  leadership  uses  the  outputs  of  modernization  planning  to  assist  them  in  the  difficult 
task  of  resource  allocation.  The  limited  funding  available  for  modernization  increases  the 
importance  of  producing  sound  modernization  plans.  An  overview  of  the  relationship 
between  the  MPP  and  AFSP  is  provided  in  Figure  1. 


Figure  1.  AFSP  and  the  MPP  (AFI  10-601, 1999) 
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Air  Combat  Command  (ACC),  the  largest  of  the  Air  Force  major  commands 
(MAJCOMs),  bears  the  responsibility  of  modernization  planning  for  the  Combat  Air 
Forces  (CAF).  The  CAF  includes  ACC,  U.S.  Air  Forces  in  Europe  (USAFE),  Pacific  Air 
Forces  (PACAF),  Air  National  Guard  (ANG),  and  the  Air  Force  Reserve  Command 
(AFRC)  (Titus,  2002b).  ACC  bears  this  responsibility  based  on  the  efficiencies  of 
centralized  modernization  planning.  This  setup  ensures  that  duplication  or  diversification 
of  modernization  efforts  is  not  encountered  between  commands. 

The  goal  of  modernization  planning  at  the  MAJCOM  level  is  the  creation  of 
Mission  Area  Plans  (MAPs)  for  specific  mission  areas  over  a  25-year  time  frame  (AFPD 
90-11,  2000).  Modernization  planning  is  accomplished  at  ACC  using  both  qualitative 
and  quantitative  methods.  The  planning  is  complex  and  is  heavily  reliant  on  subject 
matter  experts  (SME).  In  order  to  improve  the  ACC  MPP,  a  quantitative  decision 
analysis  tool  has  been  created. 

A  multi-objective  decision  analysis  (MOD A)  methodology  is  the  basis  for  the 
Combat  Air  Forces  Planning  and  Programming  Analytical  Tool  (CAFPPAT).  MODA,  in 
different  fonns,  is  widely  used  in  the  commercial  and  public  sectors  for  analysis  over  a 
wide  variety  of  decision  making  scenarios.  MODA  is  a  way  to  add  structure,  objectivity, 
and  repeatability  to  complex  decisions  (Chambal,  2002).  These  properties  make  this 
methodology  appropriate  for  the  task  of  modernization  planning  based  on  the  need  to 
optimize  the  allocation  of  scarce  resources. 

The  annual  modernization  budget  for  ACC  exceeds  $10  billion.  However, 
approximately  eighty  percent  is  fenced  for  development  of  new  aircraft,  leaving 
significantly  less  to  upgrade  current  systems  and  develop  new  weapons  and  platforms 
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(Hickman,  2002b).  The  relatively  small  portion  of  modernization  funding  left  over  is 
primarily  used  to  address  the  capability  improvements  deemed  necessary  for  aging  legacy 
systems.  There  are  over  400  programs  and  solutions  that  compete  for  the  remaining 
modernization  funding.  This  makes  selecting  the  solutions  which  provide  the  largest 
increase  in  capability  very  important. 

The  CAFPPAT  is  an  analytical  model  which  produces  results  that  are  intended  to 
assist  decision  makers.  The  outputs  generated  by  the  CAFPPAT  are  not  all  inclusive 
answers  for  decision  makers  to  rely  on  in  modernization  planning.  The  CAFPPAT  is  a 
decision  support  tool  to  help  leadership  make  modernization  planning  decisions 
(Sullivan,  2002).  The  results  of  the  CAFPPAT,  combined  with  qualitative  analysis,  can 
assist  decision  makers  in  increasing  the  effectiveness  of  the  MPP. 

CAFPPAT  is  a  hierarchical  value  model  based  on  scenarios  created  from  defense 
guidance  and  the  current  capabilities  of  the  CAF.  The  model  is  created  to  address  real 
world,  future  scenarios  where  the  combined  capabilities  of  the  CAF  would  be  utilized. 
The  model  separately  considers  the  campaign  and  system  levels  which  allow  the 
identification  of  the  tasks  necessary,  at  each  level,  to  achieve  the  desired  effects.  The 
model  provides  a  construct  over  which  potential  solutions  can  be  judged  in  regards  to  the 
mitigation  of  capability  shortfalls. 

The  outputs  of  the  CAFPPAT  are  detailed  modernization  planning  and 
programming  scenarios,  baseline  platform  definition,  campaign  level  capability 
shortfalls,  system  level  capability  shortfalls,  and  the  degree  to  which  modernization 
solutions  mitigate  capability  shortfalls  (Hickman,  2002a).  The  outputs  represent 
quantitatively  based,  objective  information  that  can  be  provided  to  decision  makers. 
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The  construction  of  this  model  is  accomplished  by  using  the  subjective  opinion  of 
SME.  The  CAFPPAT  utilizes  campaign  planners,  operators,  and  various  other  functional 
personnel  to  construct  the  model.  Effective  and  efficient  group  decision  making  is 
fundamental  in  building  and  using  this  model  for  modernization  planning. 

There  are  numerous  methodologies  available  for  use  in  facilitating  a  decision 
from  a  group  of  SME.  The  application  of  these  methodologies  is  often  situation  and 
group  specific.  Certain  methodologies  yield  better  results  based  on  group  composition  or 
the  personalities  involved.  The  overall  goal  or  focus  of  the  group  decision  can  dictate  the 
use  of  a  certain  methodology. 

Research  Scope 

The  focus  of  this  research  centers  on  the  group  decision  making  process  utilized 
for  the  campaign  level  of  the  CAFPPAT.  The  structuring  and  weighting  of  campaign 
level  tasks  will  be  observed  to  provide  the  necessary  data  for  analysis.  The  campaign 
level  is  one  component  of  a  large,  complex  modernization  model.  The  other  levels  of  the 
model  are  candidates  for  future  research  efforts. 

Research  Objectives 

This  research  effort  consists  of  three  objectives  which  provide  a  basis  for 
understanding  the  CAFPPAT  and  the  role  of  group  decision  making  in  decision  analysis. 
The  three  objectives  are  as  follows: 

1.  Analyze,  generalize  and  document  the  CAFPPAT. 

2.  Based  on  literature,  develop  a  comprehensive  taxonomy  for  group  decision 
making. 
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3.  Observe,  document,  and  analyze  the  group  decision  making  process  utilized 
for  the  CAFPPAT  campaign  level  and  provide  insight. 

Potential  Benefits 

The  potential  benefits  of  this  research  are  two-fold.  The  documentation  of  the 
CAFPPAT  will  provide  a  reference  for  the  Air  Force  and  other  DoD  organizations. 

When  compared  to  the  group  decision  making  taxonomy,  the  insight  into  the  CAFPPAT 
campaign  level  group  decision  process  will  provide  a  benchmark  for  others  to  follow. 

This  will  assist  other  organizations  in  utilizing  groups  of  SME  for  their  decision  analysis 
tools.  This  can  translate  into  more  robust  decisions  being  made,  maximizing  the 
capability  obtained  with  the  limited  funding  available. 

Additionally,  identification  of  areas  for  improvement  in  the  campaign  level 
process  will  allow  ACC  to  further  enhance  their  decision  analysis  tool.  This 
improvement  can  increase  the  fidelity  of  information  provided  to  other  planners, 
programmers,  and  decision  makers  responsible  for  modernizing  the  CAF.  A  visual 
summary  of  these  benefits  is  provided  in  Figure  2.  The  waterfall  impact  of  improving  the 
utilization  of  group  expertise  can  ultimately  affect  the  war  fighting  ability  of  the  Air 
Force  and  DoD. 
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Figure  2.  Potential  Benefits 

Thesis  Overview 

Chapter  II  is  a  literature  review  which  expands  on  the  topic  of  modernization  by 
providing  an  overview  of  the  Air  Force  process.  The  CAFPPAT  is  discussed  in  detail  to 
outline  its  creation  and  use  for  modernization  planning  within  ACC.  Finally,  a  review  of 
group  decision  making  methodologies  is  presented.  This  review  discusses  the  origins  of 
the  different  methodologies  and  outlines  the  general  process  of  each.  Chapter  III 
provides  an  in  depth  review  of  the  methodologies  applicable  to  MODA.  This  review 
identifies  the  advantages,  disadvantages,  and  the  appropriate  use  of  the  identified 
methodologies.  This  review  culminates  in  a  methodology  table  that  can  be  used  to 
identify  appropriate  methods  based  on  group  criteria. 
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Chapter  IV  documents  an  observed  campaign  level  group  decision  making 
process.  Analysis  of  this  observed  process  is  compared  to  the  methodology  table  created 
in  Chapter  III.  This  analysis  provides  insight  into  the  current  process  utilized  by  ACC. 
Chapter  V  reviews  the  methodology  constructed  and  its  use  in  decision  analysis. 
Conclusions  and  recommendations  are  provided  with  suggestions  for  further  research 
regarding  this  topic. 
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II.  Literature  Review 


Chapter  Overview 

This  chapter  provides  an  overview  of  the  current  Air  Force  modernization  effort 
and  examines  a  model  created  by  ACC  to  assist  in  modernization  planning.  Group 
decision  making  is  fundamental  to  the  use  of  this  model  and  a  comprehensive  review  of 
applicable  methodologies  is  presented. 

Transformation  and  Modernization 

In  the  last  decade,  the  issue  of  transformation  has  grown  in  importance  within  the 
DoD.  The  end  of  the  cold  war  and  the  emergence  of  smaller,  but  very  capable,  threats  to 
national  security  has  dictated  a  review  of  how  the  DoD  accomplishes  its  mission.  The 
result  of  this  assessment  is  the  need  for  transformation  to  a  force  better  suited  to  fight  and 
win  against  current  and  future  threats. 

The  Air  Force  is  engaged  in  transfonnation  efforts  by  attempting  to  reshape  the 
force  into  a  “light,  lean,  and  lethal  expeditionary  force”  (Aguilar,  2002).  The  creation  of 
the  Expeditionary  Aerospace  Force  (EAF)  is  one  example  of  Air  Force  transformation. 
The  transfonnation  effort  is  conducted  through  organizational  changes,  revised  concepts 
of  operations  (CONOPS),  and  advanced  technologies  (Deptula,  2001).  Modernization 
yields  advanced  technologies  and,  as  previously  defined,  is  the  planned  increase  of  Air 
Force  technical  sophistication.  This  increase  in  technology  is  a  critical  component  of  the 
overall  transformation  effort. 

The  focus  of  transformation  and  modernization  is  the  capabilities  that  will  be 
required  to  fight  in  future  conflicts.  The  goal  is  to  improve  the  capabilities  of  the  Air 
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Force  to  ensure  desired  effects  can  be  obtained.  The  two  focus  areas  for  modernization 


and  transformation,  in  regards  to  war  fighting  equipment,  are  maintaining  current  legacy 
systems  and  developing  new  systems.  Capabilities  are  maintained  and  increased  through 
sustaining  and  improving  the  legacy  systems.  Increased  efficiency  and  new  capabilities 
are  achieved  through  development  of  new  systems.  The  other  elements  of 
transformation,  organizational  changes  and  improved  CONOPS,  also  help  to  improve 
capabilities.  This  relationship  is  depicted  graphically  in  Figure  3. 

Capability  Improvement 

4 _ 

Modernization  &  Transformation 

i  I  j. 

Legacy  Systems 

New  Systems 

- ► 

Time 

Figure  3.  Capability  Improvement  (Hickman,  2002c) 

Air  Force  modernization  is  intended  to  be  time  phased  and  balanced  in  its 
application.  It  is  time  phased  in  terms  of  providing  the  required  capability  in  congruence 
with  the  phasing  out  of  legacy  systems.  It  is  balanced  in  regards  to  investing  across  all  of 
the  Air  Force  competencies  (Air  Force  Handbook,  2002). 
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The  capabilities  required  for  success  in  future  conflicts  are  defined  in  detail  in  the 
AFSP,  Volume  3.  These  are  defined  as  critical  future  capabilities  and  are  addressed 
through  the  combination  of  vision,  CONOPS,  innovation,  and  the  MPP.  This  highlights 
the  fact  that  Air  Force  planning  is  driven  by  the  need  to  obtain  these  critical  future 
capabilities.  This  relationship  between  capabilities  and  contributing  factors  is  presented 
in  Figure  4. 


Figure  4.  Critical  Future  Capabilities  (AFSP,  Vol.  3,  2000) 

Modernization  planning  is  a  key  component  of  the  Planning,  Programming,  and 
Budgeting  System  (PPBS)  utilized  by  the  DoD.  The  goal  of  the  PPBS  is  “to  provide  the 
best  mix  of  forces,  equipment,  and  support  attainable  within  fiscal  constraints”  (PPBS 
Primer,  1999:2).  The  PPBS  was  created  in  the  1960s  by  Secretary  of  Defense 
McNamara  in  an  attempt  to  link  the  planning  and  budgeting  efforts  of  the  DoD  through 
comprehensive  planning. 
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This  dynamic,  iterative  process  has  a  two-year  cycle,  but  consists  of  many  phases 
that  overlap.  Each  cycle  starts  with  national  defense  policy  and  culminates  in  a  budget 
submission  by  the  President  to  Congress  (PPBS  Primer,  1999).  Modernization  planning 
is  a  continuous  effort  that  is  utilized  to  produce  inputs  for  the  formulation  of  the  Program 
Objective  Memorandum  (POM).  The  POM  is  the  culmination  of  the  planning  and 
programming  phases  of  the  PPBS.  Figure  5  provides  an  overview  of  the  major 
components  of  the  PPBS. 
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Figure  5.  The  PPBS  (PPBS  Primer,  1999) 

Modernization  is  a  difficult  task  to  accomplish  due  to  an  environment  of 
constrained  resources.  There  has  been  a  steady  decline  in  DoD  and  Air  Force 
modernization  funding  since  the  mid-1980s  (Ellet,  1998).  The  issue  of  limited  funding 
dictates  that  the  Air  Force  successfully  leverages  technology  to  realize  capability 
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improvements  (Aguilar,  2002).  This  creates  the  demand  for  comprehensive  planning  that 
maximizes  the  capability  provided  to  the  end  user  for  the  money  spent. 

The  focus  on  capabilities  with  regards  to  modernization  and  transfonnation  stems 
from  the  effort  to  reengineer  the  Air  Force  Resource  Allocation  Process  (AFRAP).  The 
AFRAP  reengineering  effort  centers  on  the  creation  of  a  capabilities  framework  that  will 
be  used  in  modernization  planning  and  resource  allocation  decisions  (Lorenz,  2001b). 
This  is  a  dynamic  process  intended  to  shape  modernization  efforts  Air  Force  wide  in  the 
future. 

One  attempt  at  AFRAP  transformation  is  the  ongoing  creation  and  use  of  the  Task 
Force  CONOPS  (Stevenson,  2002).  The  creation  of  task  forces  corresponding  to  Air 
Force  mission  areas  is  intended  to  develop  capability  sets  for  each  area.  The  vision  of  the 
Air  Force  Chief  of  Staff  is  having  capabilities  driving  the  budgeting  process  (Stevenson, 
2002). 

Modernization  planning  is  one  part  of  the  overall  Air  Force  strategic  planning 
process.  Figure  6  shows  a  more  in  depth  look  at  the  relationship  between  the  MPP  and 
AFSP.  One  important  aspect  of  current  MPP  that  is  displayed  in  Figure  6  is  the  role  of 
the  MAJCOM.  The  planning  process  is  decentralized  with  most  of  the  planning  being 
conducted  primarily  at  the  MAJCOM  level  (Eidsaune,  2000).  The  MAJCOM  planning 
inputs  are  combined  to  fonnulate  the  USAF  budget  submission.  MAJCOMs  utilize  a 
standard  Air  Force  modernization  planning  process. 
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Figure  6.  The  Air  Force  Strategic  Planning  Process  (AFSP,  Vol.  3,  2000) 

The  Air  Force  Modernization  Planning  Process  (MPP)  consists  of  three  steps  that 
generate  outputs  used  to  produce  modernization  plans.  The  three  steps  are  the  Mission 
Area  Assessment  (MAA),  Mission  Needs  Analysis  (MNA),  and  the  Mission  Solution 
Analysis  (MSA).  These  three  steps  facilitate  the  creation  of  Mission  Area  Plans  (MAPs) 
and  Mission  Support  Plans  (MSPs).  The  MPP  process  is  conducted  over  a  two  year 
period  allowing  synchronization  with  the  PPBS  (AFI  10-601,  1999).  The  process  is 
conducted  by  Mission  Area  Teams  (MATs)  located  at  MAJCOMs,  Field  Operating 
Agencies,  and  Direct  Reporting  Units  responsible  for  modernization  planning. 

The  first  step,  MAA,  consists  of  transforming  military  strategy  and  guidance  into 
tasks.  These  tasks  are  determined  to  be  necessary  to  accomplish  the  prescribed  military 
objectives.  The  next  step,  MNA,  evaluates  the  ability  of  the  current  force  to  accomplish 
the  tasks  identified  during  the  MAA.  This  step  results  in  a  list  of  capability  shortfalls  to 
be  addressed  through  modernization  solutions.  The  final  step,  MSA,  identifies  potential 
material  solutions  intended  to  fix  the  capability  shortfalls.  A  material  solution  is 
identified  as  something  other  than  a  change  in  tactics,  doctrine,  training,  or  strategy  (AFI 
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10-601,  1999).  The  MSA  step  yields  a  list  of  prioritized  solutions  that  address  the 
capability  shortfalls  identified. 

The  results  of  the  first  three  steps  of  the  MPP  support  the  creation  of  MAPs  and 
MSPs.  These  plans  act  as  a  “modernization  roadmap”  for  the  next  25  years  (AFI  10-601, 
1999).  Specific  fighter  and  bomber  roadmaps  are  produced  in  addition  to  the  mission 
area  roadmaps.  All  of  these  modernization  plans  flow  into  the  Air  Force  Program 
Projection  (AFPP)  which  serves  as  a  mid  and  long  term  investment  plan.  A  depiction  of 
the  MPP,  to  include  inputs,  outputs,  and  process  flow,  is  provided  in  Figure  7. 


Modernization  Planning  Process  (MPP) 


HQ  Non-materiel  Solutions: 

Guidance  Tactics,  Doctrine, 


Figure  7.  MPP  (AFI  10-601, 1999) 

ACC  utilizes  the  Air  Force  MPP  to  identify  and  address  capability  shortfalls  for 
the  combined  Combat  Air  Forces  (CAF).  “ACC  is  the  lead  for  the  modernization  of  all 
fighter,  bomber,  search  and  rescue,  and  non-space  Command  and  Control  (C2)  and 
Intelligence,  Surveillance  and  Reconnaissance  (ISR)  forces”  (ACC  Strategic  Plan, 
2002:8).  A  detailed  depiction  of  the  ACC  MPP,  to  include  the  sources  of  guidance, 
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mission  considerations,  and  feedback  sources  is  provided  in  Figure  8.  ACC  conducts 
both  qualitative  and  quantitative  analysis  during  the  MPP  development.  One  quantitative 
method  that  is  utilized  by  ACC  is  multi-objective  decision  analysis. 


Figure  8.  ACC  MPP  (Sullivan,  2002) 


Multi-Objective  Decision  Analysis 

Multi-Objective  Decision  Analysis  (MOD A)  is  a  method  of  providing  structure  to 
complex  decision  problems  with  multiple  evaluation  criteria.  The  application  of  MOD  A 
can  provide  quantitative  decision  tools  that  are  defendable,  repeatable,  and  objective 
(Chambal,  2002).  Numerous  methods  of  conducting  MODA  have  been  developed. 
Different  MODA  approaches  include  Value  Focused  Thinking  (VFT),  Multi-Attribute 
Utility  Theory  (MAUT),  the  Analytic  Hierarchy  Process  (AHP),  and  Multiple  Criteria 
Decision  Making  (MCDM)  (Kirkwood,  1997).  All  of  these  techniques  attempt  to  provide 
structure  to  decision  problems  in  their  application. 
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The  techniques  differ  in  their  means,  but  all  result  in  information  intended  to 
assist  decision  makers.  These  techniques  provide  insight  and  analysis,  not  full  proof 
answers  to  decision  problems.  The  structure,  repeatability,  and  objectivity  of  these 
techniques  make  them  ideal  for  use  in  modernization  planning  and  the  AFRAP. 

Modernization  Utilizing  MODA 

The  use  of  MODA  for  the  purpose  of  modernization  planning  has  become  wide 
spread  throughout  the  Air  Force  and  DoD.  There  are  numerous  examples  that 
demonstrate  the  applicability  of  MODA  techniques  to  the  difficult  resource  allocations 
problems  that  face  defense  leaders.  The  examples  that  follow  do  not  represent  a 
comprehensive  list,  but  rather  a  few  notable  uses  of  MODA  in  the  DoD. 

One  popular  application  is  the  Foundations  2025  value  focused  thinking  model 
created  to  evaluate  air  and  space  dominance  in  the  year  2025  (Parnell,  1998).  In  regards 
to  modernization,  this  model  was  used  to  evaluate  futuristic  system  concepts  and 
technologies.  This  model  development  provided  an  example  for  the  use  of  MODA 
towards  complex  defense  decisions. 

In  response  to  the  AFRAP  initiative,  USAFE  created  a  resource  allocation  model 
(RAM)  utilizing  MODA  techniques.  The  USAFE  RAM  is  a  capabilities  based  model 
intended  to  “link  resource  allocation  to  strategic  planning  and  performance  management” 
(Lorenz,  2001a).  The  USAFE  RAM  utilizes  a  hierarchical  structure  consisting  of 
capabilities,  mission  essential  tasks,  programs,  and  measures  (Lorenz,  2001a).  This 
MODA  tool  is  intended  to  allow  a  decision-maker  the  ability  to  balance  capabilities  over 
time  (Lorenz,  2001a). 
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The  Army  conducted  modernization  planning  for  the  utility  helicopter  fleet  by 
using  MOD  A  techniques.  The  focus  was  to  maintain  capability  by  integrating  MOD  A 
techniques  with  general  qualitative  methods  and  cost  analysis.  MODA  was  one  part  of 
an  analysis  that  included  platfonn  evaluation  and  fleet  mixture  analysis  resulting  in  a 
fleet  modernization  strategy  and  implementation  plan  (Prueitt,  2000). 

MODA  was  utilized  by  the  Air  Force  Studies  and  Analysis  Agency  (AFSAA)  to 
conduct  an  analysis  of  alternatives  for  a  next  generation  of  gunship.  A  value  focused 
thinking  model  was  created  with  the  goal  of  screening  alternatives  that  could  replace  the 
current  AC- 130  Gunship  (Renfro,  2002).  This  information  was  utilized  in  congruence 
with  infonnation  provided  by  an  independent  contractor  to  evaluate  the  most  appropriate 
alternatives  for  future  investment. 

In  1999,  a  joint  sponsored  effort  by  ACC  and  Air  Force  Space  Command 
(AFSPC)  was  undertaken  to  improve  the  implementation  of  the  MPP.  This  effort  was 
labeled  the  Aerospace  Integrated  Investment  Study  (ASUS)  and  included  a  MODA  model 
to  address  a  standardized  capability  framework  (ASUS,  2000).  “The  primary  purposes  of 
ASUS  are  to  standardize  the  analysis  used  to  implement  the  MPP  at  ACC  and  AFSPC 
and  to  rectify  deficiencies  in  their  existing  analytical  approaches”  (ASUS,  2000: 1).  This 
effort  resulted  in  an  initial  model,  based  on  capabilities,  providing  an  example  for  both 
ACC  and  AFSPC  to  follow. 

AFSPC  has  incorporated  the  results  of  ASUS  into  an  integrated  planning  process 
(IPP)  which  “underpins  AFSPC’s  responsibilities  to  equip  the  Air  Force  with  the  space 
portion  of  aerospace  power”  (Space  IPP  Handbook,  2000).  AFPSC  utilizes  a  value 
model  to  evaluate  capabilities  as  part  of  the  overall  IPP.  Analogous  to  the  AFSPC  effort, 
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ACC  has  applied  MODA  to  modernization  planning  in  attempt  to  improve  resource 
allocation. 

Combat  Air  Forces  Planning  and  Programming  Analytical  Tool 

The  ACC  Directorate  of  Requirements  (DR),  Analysis  Division  (DRY),  Resource 
Analysis  Branch  (DRYR),  is  charged  with  providing  CAF  senior  leadership  with  an 
objective,  analytically  developed  assessment  of  operational  and  tactical  level  capabilities 
and  shortfalls.  This  assessment  is  to  be  used  to  evaluate  operational  and  tactical 
capabilities,  solution  to  needs  analysis,  and  the  return  on  investment  of  solutions 
(ACC/DRYR  web  page,  2002).  This  fits  into  the  mission  of  ACC/DR  providing  “better 
definition  for  modernization  and  sustainment  of  weapons  systems”  (ACC/DR  Goal  web 
page,  2002).  In  an  attempt  to  meet  this  requirement,  ACC/DRYR,  in  a  combined  effort 
with  ACC/DRPX,  created  the  CAFPPAT. 

The  CAFPPAT  is  a  MODA  tool  intended  to  assist  in  difficult  modernization 
decisions.  “It  is  a  tool  that  is  applicable  for  a  given  period  of  time  across  a  given  set  of 
scenarios  to  help  achieve  a  desired  set  of  effects  using  improvements  or  modifications  to 
a  baseline  set  of  capabilities”  (Titus,  2002a:2).  The  key  components  of  this  model  are  the 
scenarios,  campaign  level  capabilities,  and  aircraft/system  level  capabilities. 

These  components  are  arranged  in  a  hierarchical  model  using  the  multi-attribute 
utility  theory  form  of  MODA.  The  result  of  this  hierarchical  structure  of  components  is  a 
score  for  material  solutions  in  terms  of  mitigating  capability  shortfalls.  An  overview  of 
the  CAFPPAT  is  provided  in  Figure  9.  Specific  levels  of  this  hierarchical  model  will  be 
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addressed  in  the  following  sections  and  an  example  of  how  a  solution  flows  through  the 
model  will  be  diagramed. 
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Figure  9.  CAFPPAT  Overview  (Flickman,  2002b) 

The  model  follows  the  MPP  in  assessing  the  capability  shortfalls  of  the  CAF  and 
the  scoring  of  potential  solutions.  “The  primary  goal  of  CAFPPAT  in  ACC/DR  is  to 
provide  the  analytical  underpinnings  for  the  MAA,  MNA,  and  MSA,  which  feed  the 
creations  of  MAPs  for  specific  mission  areas  over  a  25-year  period”  (Titus,  2002a:5). 
Figure  10  displays  the  CAFPPAT  process  broken  into  seven  steps  and  how  each  part  fits 
into  the  MPP  structure. 
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Figure  10.  CAFPPAT  Process  (Flickman,  2002b) 


The  CAFPPAT  represents  three  dimensional  planning  and  programming  based  on 
capabilities,  systems,  and  threats.  Capability  based  planning  cannot  function  independent 
of  the  systems  that  provide  the  capabilities  or  the  threats  the  capabilities  are  required  to 
counter.  These  three  dimensions  must  be  considered  and  are  not  mutually  exclusive  in 
modernization  planning  (Hickman,  2002b).  One  application  of  the  CAFPPAT  has  been 
completed  by  ACC/DRYR  and  ACC/DRPX  in  support  of  the  Fiscal  Year  (FY)  2004 
POM.  The  following  sections  outline  the  CAFPPAT  components  and  process  as  they 
were  developed  for  the  FY2004  POM  application.  The  CAFPPAT  is  a  dynamic  tool  that 
is  continually  being  improved  to  better  satisfy  its  intended  use. 
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Model  Assumptions. 

In  the  application  of  MODA,  numerous  assumptions  are  made  to  facilitate  the 
creation  and  use  of  decision  models.  The  most  critical  assumptions  made  for  the 
CAFPPAT  deal  with  force  structure,  scenario  coverage,  and  capabilities.  One 
assumption  is  that  an  expeditionary  force  is  necessary  to  participate  in  the  scenarios 
chosen  (Sullivan,  2002).  The  chosen  scenarios  are  assumed  to  accurately  portray  the 
most  probable  future  conflicts  and  needed  capabilities.  Finally,  the  hierarchies  created 
for  the  model  are  assumed  to  be  collectively  exhaustive,  mutually  exclusive, 
preferentially  independent,  and  contain  the  minimum  number  of  objectives  possible 
(Sullivan,  2002).  These  assumptions  are  made  based  on  extensive  research  and 
coordination  with  SME. 

Scenario  Definition. 

The  first  step  in  utilizing  the  CAFPPAT  is  to  define  a  set  of  scenarios.  These 
scenarios  provide  the  basis  for  detennining  what  tasks  the  CAF  need  to  accomplish  in 
order  to  achieve  the  desired  effects.  Scenarios  are  developed  based  on  Defense  Planning 
Guidance  (DPG)  and  other  sources  of  national  military  strategy.  Another  contributing 
factor  in  building  scenarios  is  the  availability  of  data  from  sources  such  as  the 
Modernized  Integrated  Database  (MIDB).  The  data  is  critical  because  it  increases  the 
objectivity  and  accuracy  of  the  model.  The  scenarios  chosen  for  use  in  the  CAFPPAT 
can  occur  anytime  during  the  next  25  years  and  can  be  any  type  of  known  conflict. 

Once  a  time  frame  and  type  of  conflict  is  chosen,  19  different  variables  are 
considered  to  further  define  the  scenarios.  These  variables  represent  likely  conditions 
and  are  necessary  to  assess  the  operating  environment.  Once  a  type  of  conflict  and 
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timeframe  has  been  chosen,  the  19  variables  remaining  result  in  over  3.8  billion  possible 
scenarios.  This  demonstrates  the  ability  of  CAFPPAT  to  capture  the  specifics  involved  in 
a  possible  conflict. 

The  first  application  of  the  CAFPPAT  utilized  four  scenarios.  These  scenarios 
were  chosen  based  on  their  congruence  with  the  DPG  and  availability  of  data  (Sullivan, 
2002).  These  four  scenarios  represent  a  sample  of  probable  conflicts  based  on  DPG  and 
national  military  strategy.  The  credibility  of  the  CAFPPAT  is  increased  with  the 
inclusion  of  more  scenarios,  such  as  homeland  defense,  in  future  applications  of  the 
model  (Hickman,  2002b). 

The  four  scenarios  chosen  represent  the  top  branch  of  the  hierarchical  MODA 
model.  The  scenarios  are  weighted  equally  based  on  the  assumption  of  the  model 
designers.  The  assumption  is  that  the  CAF  need  to  modernize  for  each  of  these  scenarios 
is  equivalent  (Titus,  2002a).  Each  scenario  will  have  its  own  hierarchical  structure 
representing  needed  capabilities  and  CAF  ability  to  achieve  the  necessary  effects.  This 
concept  is  displayed  in  Figure  1 1  with  notional  scenario  examples.  Once  again,  each 
scenario  represents  one  branch  of  the  MODA  model. 
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Solution  List 


Scenario  1 

Scenario  2 

Scenario  3 

Scenario  4 

Small  Scale  Contingency  -  2009 

Small  Scale  Contingency  -  2007 
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Figure  11.  CAFPPAT  Scenario  Structure 
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Campaign  Level  Capabilities. 

Campaign  level  capabilities  represent  the  effects  that  need  to  be  achieved  for  each 
scenario.  CAFPPAT  is  designed  to  address  seven  different  mission  areas  that  ACC  is 
responsible  for  modernizing.  These  mission  areas  directly  correspond  to  four  analytical 
capability  areas  that  were  created  by  the  CAFPPAT  designers.  The  capability  and 
corresponding  mission  areas  are  displayed  in  Table  1. 


Table  1.  Capability  and  Mission  Areas 


Capability  Area 

Mission  Area 

Neutralization 

Global  Attack 

Air  Superiority 
Information  Warfare 

Force  Protection/Infrastructure/Logistics  (FP/INF/LOG) 

Agile  Combat  Support 
Information  Warfare 

Combat  Search  and  Rescue  (CSAR) 

CSAR 

Intelligence,  Surveillance,  and  Reconnaissance  (ISR) 

ISR 

Command  and  Control 

C~ 

Task  lists  have  been  created  for  all  live  of  the  campaign  level  capability  areas. 
The  five  task  lists  represent  the  specific  tasks  that  need  to  be  accomplished  to  achieve 
success  in  that  particular  mission  area.  Each  task  list  is  a  hierarchical  structure  providing 
the  appropriate  level  of  detail  such  that  the  tasks  can  be  understood  and  compared 
(Hickman,  2002b).  There  are  separate  task  lists  for  neutralization,  combat  search  and 
rescue  (CSAR),  force  protection/infrastructure/logistics,  command  and  control,  and 
intelligence/surveillance/reconnaissance  (ISR).  The  task  lists  were  created  from  Volume 
3  of  the  AFSP,  existing  operation  plans,  CONOPS,  and  Air  Force  and  Joint  task  lists 
(Sullivan,  2002).  During  the  first  application  of  the  CAFPPAT,  no  task  list  for  ISR  was 
created.  This  task  list  has  since  been  created  and  is  in  the  process  of  verification  and 
validation. 
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The  lowest  level  of  each  capability  task  list  is  represented  by  evaluation  measures. 
These  measurable  tasks  are  the  foundation  for  the  MODA  techniques  utilized  in  each  of 
these  capability  hierarchies.  Table  2  provides  a  list  of  the  capability  hierarchies  and  their 
respective  number  of  measurable  tasks.  This  provides  insight  into  the  large  number  of 
capability  tasks  that  are  addressed  in  the  CAFPPAT. 


Table  2.  Capability  Evaluation  Measures 


Capability  Task  List 

Evaluation  Measures 

Neutralization 

151 

Combat  Search  and  Rescue 

6 

Force  Protection/Infrastructure/Logistics 

309 

Intelligence/Surveillance/Reconaissance 

136 

Command  and  Control 

30 

Air  to  air  refueling  (AAR)  and  airlift  (Lift)  capabilities  are  notionally  included  in 
the  CAFPPAT.  Modernization  planning  for  these  areas  is  the  responsibility  of  Air 
Mobility  Command  (AMC).  ISR  and  Command,  Control,  Communications,  and 
Computers  (C4)  are  mission  areas  that  are  under  development  for  consideration  in  further 
applications  of  the  CAFPPAT. 

System  Level  Capabilities. 

The  system  level  of  the  CAFPPAT  represents  the  complex  man  machines,  or 
weapon  system  platforms,  that  are  used  to  accomplish  campaign  level  tasks.  “Platforms 
have  inherent  capabilities  to  accomplish  tasks.  These  capabilities  are  broken  down  into 
platform  level  tasks  and  are  displayed  in  a  hierarchical  format  similar  to  the  campaign 
level  tasks”  (Sullivan,  2002:5).  Each  platform  (aircraft)  has  a  capability  hierarchy 
consisting  of  the  following  six  task  categories;  availability,  effectiveness,  sorties, 
footprint,  survivability,  and  safety  (Sullivan,  2002).  The  system  capability  hierarchy 
used  to  evaluate  each  platform  has  168  evaluation  measures.  The  lowest  level  of  Figure 
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12  shows  the  system  level  capabilities  and  the  aircraft  that  are  applicable  to  each 
campaign  level  capability. 
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Figure  12.  System  Level  Capabilities  (Hickman,  2002b) 


CAF  Baseline  and  Contribution  Matrix 


In  order  to  determine  where  capability  shortfalls  exist,  the  current  CAF  capability 
baseline  is  determined.  The  CAF  baseline  represents  the  current  force  structure 
consisting  of  the  available  aircraft.  This  baseline  incorporates  such  factors  as  “postulated 
sub-systems,  projected  mission  capable  rates,  and  future  availability  (due  to  service  life)” 
(Hickman,  2002b). 

The  creation  of  the  campaign  level  task  list  details  what  tasks  need  to  be 
accomplished  in  the  different  mission  areas  to  achieve  the  desired  effects  for  each 
scenario.  However,  the  CAF  may  not  be  capable  of  accomplishing  all  of  the  tasks  on  the 
list.  Additionally,  the  CAF  will  be  participating  in  a  joint  effort  with  other  services  to 
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accomplish  the  tasks  for  each  scenario.  Therefore,  the  contribution  of  CAF  for  each 
scenario  and  task  list  must  be  detennined. 

A  contribution  matrix  allows  SME,  such  as  campaign  planners,  to  determine  how 
much  of  each  task  should  be  allocated  to  each  of  the  appropriate  platforms  (Titus,  2002a). 
The  CAF  baseline  provides  the  allocation  limitations  for  the  contribution  matrix.  When 
the  allocation  is  complete,  the  matrix  represents  how  the  projected  CAF  baseline  will  be 
utilized  to  achieve  the  campaign  level  tasks. 

Needs  List  Development 

The  creation  of  the  CAF  baseline  and  contribution  matrix  facilitates  the  creation 
of  a  campaign  level  needs  lists.  This  is  accomplished  through  a  weighting  process  in 
which  SME  are  utilized.  Each  measurable  task  in  each  of  the  campaign  level  capability 
areas  is  evaluated  to  identify  shortfalls.  This  is  accomplished  by  having  the  SME 
determine  how  important  each  task  is  to  improve  and  how  well  the  CAF  currently 
performs  the  task.  The  evaluation  scale  utilized  is  shown  in  Figure  13. 

A  point  estimate  is  made  on  the  scale  for  each  task  on  that  particular  level  of  the 
hierarchy.  These  point  estimates  are  then  mathematically  evaluated  to  detennine  an 
index  representing  a  quantitative  capability  shortfall.  The  index  for  each  measurable  task 
is  utilized  to  calculate  the  capability  shortfall  index  for  each  level  of  the  MODA 
hierarchy. 
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Figure  13.  Evaluation  Methodology  (Titus,  2002) 


This  mathematical  exercise  results  in  a  “need  score”  for  each  task  within  each 
hierarchy  (Sullivan,  2002).  The  prioritization  of  the  resulting  scores  results  in  a  needs 
list.  The  needs  list  displays  the  capability  shortfalls  in  prioritized  order  to  improve.  The 
five  campaign  level  needs  lists  are  then  horizontally  integrated  into  one  list  by  utilizing 
the  SME  for  each  list  (Sullivan,  2002).  Each  of  the  individual  lists  is  important  for 
analysis  purposes,  but  an  integrated  list  displays  “where  the  greatest  need  for 
improvement  lies  regardless  of  the  type  of  task”  (Sullivan,  2002). 

The  system  level  needs  list  is  created  in  the  same  manner  as  the  campaign  level 
needs  list.  The  campaign  capability  shortfalls  are  traced  to  the  systems  that  are  allocated 
to  complete  the  tasks.  The  contribution  matrix  acts  as  a  link  between  the  campaign  level 
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capability  shortfalls  and  the  systems  that  need  to  be  improved  to  mitigate  the  shortfalls. 
The  measurable  tasks  within  the  system  level  hierarchies  are  evaluated  in  the  same 
manner  with  SME  determining  the  performance  and  importance  (Sullivan,  2002).  This 
process  results  in  needs  lists  which  are  then  prioritized.  “These  needs  lists  will  be 
different  for  each  platform  due  to  the  unique  capabilities  each  bring  to  the  fight” 
(Sullivan,  2002:33).  The  system  level  needs  lists  can  be  combined  to  create  a  complete 
system  list  for  each  scenario  or  over  all  of  the  scenarios. 

Solution  Scoring 

The  scoring  of  potential  solutions,  in  terms  of  mitigating  capability  shortfalls,  is 
also  accomplished  by  SME.  The  most  common  SME  utilized  for  this  step  of  the  process 
is  the  Program  Element  Monitor  (PEM).  The  PEM  is  the  focal  point  and  primary 
proponent  for  their  particular  platfonn.  The  scoring  of  solutions  starts  at  the  platform 
level  of  the  CAFPPAT.  The  scores  that  result  represent  “a  percentage  improvement  to 
the  baseline  configuration  of  the  platform”  (Titus,  2002a). 

Any  platform  level  capability  improvement  is  mathematically  transferred  to  the 
corresponding  campaign  level  capabilities  that  it  has  a  direct  affect  upon.  This 
mathematical  roll-up  provides  a  final  score  indicating  how  much  improvement  over  the 
baseline  capability  each  solution  provides.  This  allows  solutions  to  be  compared  with 
each  other  and  prioritized.  The  solutions  given  the  highest  priorities  represent  the  largest 
mitigation  of  capability  shortfalls. 

The  flow  of  a  potential  solution  through  the  different  levels  of  the  CAFPPAT  is 
depicted  in  Figure  14.  The  solution  is  evaluated  against  applicable  platforms  which  then 
affect  certain  campaign  level  tasks.  The  effect  is  then  rolled  up  into  the  applicable 


29 


scenarios  that  rely  on  those  tasks  to  achieve  the  desired  effects.  Figure  14  also  displays 
the  capability  of  the  model  to  generate  output  at  each  of  the  levels. 
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Figure  14.  CAFPPAT  Solution  Flow 
CAFPPAT  Outputs. 

The  CAFPPAT  is  a  comprehensive  modernization  tool  that  produces  a  variety 
outputs  that  can  be  utilized  for  decision  support.  The  list  of  solutions  is  the  primary 
CAFPPAT  output: 

The  chief  output  of  the  MPP  is  a  prioritized  list  of  competing  solutions  to  satisfy 
the  needs.  Solution  lists  can  be  prioritized  within  a  platform  or  functional  area.  It 
can  span  across  platforms  to  encompass  an  entire  mission  area.  Finally,  the  MPP 
can  integrate  all  CAF  solutions  into  a  single  prioritized  solution  list.  These  lists 
can  be  prioritized  based  on  the  increase  of  capability  the  solution  brings. 

(Sullivan,  2002) 

The  flexibility  in  analyzing  the  prioritized  solution  list  allows  CAFPPAT  contribution  to 
many  different  modernization  planning  efforts. 
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Other  outputs  provided  by  the  CAFPPAT  are  a  baseline  capability  assessment  and 
lists  of  capability  shortfalls  at  the  campaign  and  system  level.  These  outputs  are  part  of 
the  CAFPPAT  process  of  scoring  solutions,  but  can  be  useful  for  analysis  on  their  own. 
Different  CAF  modernization  planners  can  utilize  these  outputs  in  their  own  analyses. 

A  solution-to-need  matrix  is  created  upon  completion  of  a  CAFPPAT  cycle  for 
each  platform.  “This  matrix  considers  the  needs  of  each  platfonn,  prioritizes  them  by  the 
“need  to  improve”  score  and  then  links  competing  solutions  to  the  needs  they  address” 
(Sullivan,  2002:49).  This  output  provides  a  way  to  evaluate  the  solution  list  to  see  if  the 
most  critical  platfonn  needs  are  being  addressed  (Sullivan,  2002).  This  matrix  is 
particularly  useful  to  the  fighter  and  bomber  roadmap  planners  due  to  the  ability  to 
analyze  each  platfonn  individually.  A  visual  summary  of  the  outputs  discussed  is 
provided  in  Figure  15. 


Figure  15.  CAFPPAT  Outputs 
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Model  Limitations. 


There  are  numerous  limitations  that  were  discovered  in  the  first  application  of  the 
CAFPPAT.  Capability  areas  not  controlled  by  ACC  were  assumed  to  be  non-limiting  in 
achieving  the  desired  effects  (Titus,  2002a).  Areas  such  as  ISR  and  AAR  could  actual 
play  a  role  in  achieving  the  desired  outcome  and  need  to  be  incorporated  in  further  uses 
of  the  model.  The  limited  number  of  scenarios  evaluated  limits  confidence  that  the 
capabilities  addressed  are  comprehensive.  The  inclusion  of  more  scenarios  will  increase 
the  fidelity  of  the  model  (Titus,  2002a). 

More  accuracy  is  needed  in  the  data  used  to  facilitate  the  CAFPPAT  process. 
Better  data  coupled  with  improved  methods  of  determining  the  current  performance  of 
tasks  will  increase  confidence  in  the  model  (Titus,  2002a).  The  sheer  size  and 
complexity  of  the  model  makes  it  difficult  to  use.  A  small  number  of  individuals  know 
the  process  and  it  consumes  a  lot  of  time  to  complete.  Efforts  to  improve  model 
efficiency  and  the  ease  in  which  it  is  created  are  currently  being  pursued. 

The  Role  of  Experts. 

A  fundamental  premise  of  the  CAFPPAT  is  the  reliance  upon  SME.  The  SME 
are  used  for  the  actual  construction  of  the  model,  task  weighting,  and  scoring  of  potential 
solutions.  ACC/DRYR  attempts  to  obtain  the  most  knowledgeable  and  experienced 
group  of  individuals  for  each  portion  of  the  model  where  SME  are  required.  In  almost  all 
cases,  it  is  more  than  one  SME  participating  in  task  list  creation,  weighting,  and  solution 
scoring. 

Groups  of  experts  are  relied  upon  for  the  CAFPPAT  due  to  the  time  and  effort 
required  to  complete  the  modeling  exercise.  The  CAFPPAT  is  intended  to  assist  senior 
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Air  Force  decision  makers  who  are  often  unable  to  devote  the  necessary  time  to  complete 
the  model  themselves.  Therefore,  SME  are  utilized  in  place  of  the  decision  makers  to 
provide  the  decision  supporting  information.  Group  decision  making  plays  a  prominent 
role  in  the  appropriate  use  and  credibility  of  the  model  and  its  results.  The  significance 
of  using  groups  for  decision  making  is  not  unique  to  the  Air  Force.  “Decision  making 
groups  are  pervasive  in  both  the  private  and  public  sectors  of  our  society”  (Seaver, 
1976:1). 

Group  Decision  Theory 

“Group  decision  covers  a  wide  range  of  collective  decision  processes  and 
encompasses  numerous  methods  designed  under  various  assumptions  and  for  different 
circumstances”  (Zahedi,  1996:265).  In  literature,  these  methods  have  been  classified  in 
different  ways  based  on  process  mechanics  or  functional  use.  Seaver  (1976)  suggests  that 
the  two  general  procedures  for  obtaining  a  group  decision  are  mathematical  aggregation 
procedures  and  behavioral  methods.  The  aggregation  procedures  utilize  mathematical 
formulas  to  reach  a  group  decision  while  the  behavioral  methods  utilize  interaction  or 
communication  (Seaver,  1976). 

Alternatively,  Srisoepardani  (1996)  classifies  group  decision  methods  based  on 
their  functional  purpose.  The  methods  are  either  utilized  for  the  structuring,  ordering  and 
ranking,  or  structuring  and  measuring.  Zahedi  (1996)  proposes  the  following  five 
categories  for  classification  of  group  decision  methods:  group  utility  analysis,  group 
consensus,  group  analytic  hierarchy  process,  social  choice  theory,  and  game  theory. 
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Finally,  Dewar  (1996)  proposes  that  the  methods  can  be  categorized  as  statistical 
group  methods,  unstructured  group  interaction,  and  structured,  direct  interaction. 
Statistical  group  methods  tabulate  group  answers  and  no  interaction  occurs.  Unstructured 
group  interaction,  also  known  as  face  to  face  communication,  results  in  an  agreed  upon 
decision.  Structured,  direct  interaction  utilizes  the  benefits  of  group  communication  and 
adds  structure  in  attempt  to  counter  any  negative  aspects  of  interaction. 

These  four  classification  systems  are  logical  and  useful  based  on  the  specific 
requirements  of  the  decision  analysis  technique  being  utilized.  However,  significant 
overlap  exists  in  the  classifications  with  each  system  representing  a  different  view  of 
many  of  the  same  methods.  This  research  effort  is  not  concerned  with  determining  which 
method  of  categorization  is  correct.  Therefore,  the  review  of  each  method  will  suggest 
the  different  classifications  that  each  method  will  fall  under. 

In  some  group  decision  literature,  creativity  techniques  are  sometimes  identified 
as  group  decision  methods.  The  brainstonning  technique  and  boundary  examination  are 
two  examples  of  idea  generation  techniques  that  have  been  labeled  group  decision 
methods.  This  research  effort  will  only  identify  and  label  group  decision  methods  that 
result  in  some  form  of  consensus  answer. 

Group  decision  methods  are  just  one  part  of  the  large  field  of  management 
science.  These  methodologies  utilize  facilitation  techniques  to  varying  degrees  based  on 
the  process  involved  to  provide  input  into  decision  analysis  techniques.  This  relationship 
is  represented  in  Figure  16  and  it  is  not  assumed  that  the  three  components  can  be 
evaluated  independent  of  each  other.  Facilitation  is  often  a  critical  part  of  the  group 
decision  methods  available  and  some  methods  are  considered  decision  analysis 
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techniques.  This  blending  of  components  is  evident  in  the  decision  analysis  literature  as 
indicated  earlier  by  the  overlapping  classification  systems. 


Group  decision  methodologies  will  be  reviewed  and  the  role  of  facilitation  will  be 
addressed  based  on  its  role  in  utilizing  each  method.  The  group  decision  methods  that 
can  be  considered  decision  analysis  techniques  will  not  be  included  in  this  review.  The 
analytic  hierarchy  process  identified  by  Zahedi  (1996)  as  a  group  decision  method,  for 
example,  is  considered  a  decision  analysis  technique  (MODA)  in  the  context  of  this 
research  effort.  The  focus  of  this  section  is  to  review  methods  that  can  provide  input  into 
decision  analysis  techniques.  Appendix  B  provides  an  exhaustive  listing  of  all  the 
techniques  and  methods  reviewed  for  this  research  effort  and  their  subsequent 
classification  for  this  study. 

Research  has  been  conducted  for  years  in  the  field  of  group  decision  theory  in  an 
attempt  to  improve  the  ability  to  consolidate  the  knowledge  and  experience  of  experts  to 
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assist  in  decision  making.  According  to  one  author,  there  are  many  potential  benefits  in 
utilizing  groups  for  complex  decisions: 

First,  groups  are  more  likely  than  individuals  to  have  a  broad  and  better  range  of 
skills  and  knowledge  pertaining  to  the  decision.  Second,  groups  provide  the 
opportunity  for  an  effective  division  of  labor  to  acquire  and  process  the  vast 
amount  of  information  needed  for  the  decision.  Third,  when  groups  are  composed 
of  members  representing  a  diversity  of  interests,  the  decision  is  perceived  to  be 
more  representative  of  the  needs  of  members.  This  results  in  wider  acceptance  of 
and  greater  commitment  to  the  decision.  (Bernard,  1995:251) 

Alternatively,  the  use  of  groups  for  decision  making  also  comes  with  issues  that 

hinder  the  decision  analysis  process.  The  issues  represent  barriers  to  the  effective  use  of 

experts  in  group  decision  settings.  Polarization,  risky  shift,  representativeness, 

availability,  anchoring  and  adjustment,  motivational  bias,  groupthink,  social  loafing,  and 

group  conflict  are  all  issues  that  can  be  labeled  group  dynamics.  These  issues  fall  under 

the  realm  of  social  psychology  (Seaver,  1976)  and  will  be  addressed  tangentially  through 

the  review  of  the  different  methods.  The  following  sections  represent  a  thorough  review 

of  current  group  decision  methods. 

Face  to  Face  Group  Interaction. 

This  method  of  group  decision  making,  in  its  simplest  form,  consists  of  a  group  of 
people  discussing  a  decision  problem  resulting  in  a  consensus  answer.  Group  conflict  or 
disagreement  is  resolved  through  discussion  and  compromise  to  reach  consensus.  This 
method  is  often  recognized  as  the  conventional  problem  solving  group.  This  is  an 
unstructured  method  which  allows  its  use  for  a  wide  variety  of  functions.  Ranking  and 
selection  between  alternatives  along  with  idea  generation  are  just  a  few  examples  of 
decision  problems  that  can  be  addressed  with  this  method.  This  method  is  the  foundation 
for  all  other  group  decision  methods  that  utilize  interaction  and  communication. 
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Voting. 

Voting  is  part  of  social  choice  theory  and  “involves  selecting  an  alternative  or 
candidate  based  on  multiple  criteria”  (Zahedi,  19996:269).  “The  social  choice  theory 
investigates  the  process  of  arriving  at  a  group  decision  in  democratic  societies  through 
the  expression  of  majority’s  will”  (Zahedi,  1996:269).  Voting  is  made  up  of  the  actual 
process  of  each  individual  choosing  an  alternative  and  the  subsequent  aggregation  to 
determine  the  best  choice.  The  different  aggregation  techniques  are  the  focus  of  this 
method  review. 

Plurality  voting  consists  of  simply  counting  the  first  place  votes  and  the 
alternative  with  the  most  is  selected.  The  Borda  rule  utilizes  the  average  rank  value  of 
each  alternative  to  determine  which  to  choose  (Chamberlin,  1985).  The  Borda  rule  is  one 
form  of  trimmed  mean  voter  aggregation.  The  average  rank  of  each  alternative  is 
computed  and  compared  for  selection.  Another  form  is  the  trimmed  median  where  for 
each  alternative,  the  median  rank  is  representative  and  utilized  for  selection  (Hurley, 
2002). 

“The  Hare  system,  also  known  as  preferential  voting,  is  a  sequential  elimination 
system”  (Chamberlin,  1985:196).  The  votes  are  counted  and  the  alternative  with  the 
majority  of  first  place  votes  is  selected.  If  this  does  not  occur  after  the  first  vote,  the 
alternative  with  the  fewest  number  of  first  place  votes  is  eliminated.  The  process  is 
repeated  until  the  one  alternative  obtains  the  majority  of  first  place  votes. 

The  Coombs  system  is  identical  to  the  Hare  system  except  for  the  methodology 
used  to  eliminate  alternatives.  Under  this  system,  the  alternative  with  the  largest  number 
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of  last  place  votes  is  eliminated.  This  is  done  until  an  alternative  carries  the  majority  of 
first  place  votes  (Chamberlin,  1985). 

Approval  voting  consists  of  each  voter  casting  either  a  one  or  zero  for  each 
alternative.  An  alternative  given  a  one  shows  approval  where  a  zero  shows  that  the 
alternative  does  not  measure  up  for  that  particular  voter.  The  approval  votes  are  counted 
and  the  alternative  with  the  most  approvals  is  selected  (Nurmi,  1984). 

Finally,  cumulative  voting  allows  each  voter  to  cast  a  certain  number  of  votes. 

The  voter  can  allocate  votes  between  alternatives  in  any  combination  to  indicate 
preference  in  terms  of  magnitude  (Brams,  2002).  The  end  result  is  a  ranking  of 
candidates  upon  tabulation  of  the  votes. 

Delphi  Technique. 

The  Delphi  Technique  is  a  structured,  iterative  process  that  focuses  on  the  goal  of 
obtaining  group  consensus.  Delphi  was  created  by  the  RAND  Corporation  in  the  1950s 
as  a  forecasting  tool.  The  intent  of  creating  this  procedure  was  to  reduce  the  negative 
aspects  of  face  to  face  group  interaction  (Dalkey,  1967).  Four  characteristics  are 
fundamental  to  the  Delphi  method  and  its  successful  implementation.  The  foundations  of 
this  method  are  anonymity,  iteration,  controlled  feedback,  and  statistical  aggregation  of 
group  response  (Rowe,  1999). 

According  to  Zahedi,  (1996),  there  are  three  steps  to  the  Delphi  process  to  reach  a 
group  consensus  answer.  The  first  step  is  to  design  a  survey  that  addresses  the  problem 
and  then  have  each  group  member  respond  individually  to  the  survey  as  the  second  step. 
The  third  step  of  the  process  is  to  analyze  the  results,  adjust  the  survey  where  needed,  and 


38 


provide  aggregate  results  to  each  individual  for  their  feedback.  A  sufficient  number  of 
iterations  of  these  three  steps  should  be  completed  until  a  consensus  answer  is  reached. 

“This  widespread  use  of  the  Delphi  Technique  has  led  to  many  variations  in 
format  and  implementation  among  practitioners”  (Erffmeyer,  1986: 121).  According  to 
Rowe  (1999),  a  classical  form  of  Delphi  consists  of  the  first  round  being  unstructured 
with  inputs  regarding  the  problem  being  solicited  from  experts.  The  following  rounds  are 
structured  with  statistical  feedback  being  provided  to  the  participants. 

Common  Delphi  utilization  has  the  first  round  of  the  procedure  structured  to  make 
it  easier  on  the  facilitator.  The  number  of  rounds  used  is  variable,  but  the  total  rarely 
exceeds  two  (Rowe,  1999).  The  optimal  number  of  rounds  to  be  conducted  was 
researched  by  Erffmeyer  (1986)  and  concluded  to  be  four  rounds.  Additionally,  the 
experts  are  usually  only  tasked  to  provide  one  statistic  as  an  input  into  the  process  for 
simplification  purposes  (Rowe,  1999). 

Nominal  Group  Technique. 

The  Nominal  Group  Technique  is  another  structured  group  decision  method  that 
was  created  by  Delbecq  and  Van  de  Ven  in  1971  (Zahedi,  1996).  This  method  is  similar 
to  the  Delphi  method  in  attempts  to  utilize  the  benefits  of  collective  group  expertise  while 
countering  the  negative  aspects  of  group  interaction.  “The  object  of  the  method  is  to 
offer  a  non-conflictual  process  for  arriving  at  creative,  non-routine  decisions”  (Mahler, 
1987:337).  Contrary  to  the  Delphi  method,  anonymity  is  not  part  of  the  process  and 
group  members  are  assembled  together.  Additionally,  this  method  of  group  decision 
making  requires  an  active  leader  (Seaver,  1978). 
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In  its  original  form,  this  technique  consists  of  six  steps  that  follow  the  clear 
definition  of  the  problem  being  presented  to  the  group  (Bartunek,  1984).  The  first  step  is 
to  have  each  group  member  silently  generate  ideas  and  write  then  down.  The  second  step 
is  to  have  the  facilitator  record  all  of  the  ideas  in  a  round  robin  fashion  until  all  are  listed. 
Third,  each  idea  is  discussed  until  everyone  in  the  group  understands  them  fully. 

The  fourth  step  requires  each  group  member  to  secretly  rate  the  alternatives  and 
the  facilitator  records  the  votes  for  group  review.  The  group  then  discusses  the  vote  just 
taken  to  further  clarify  issues.  The  sixth  and  final  step  consists  of  iterations  of  secret 
votes  and  discussion  until  a  clear  choice  is  identified  (Bartunek,  1984). 

Group  Utility  Analysis. 

This  is  a  statistical  method  used  to  mathematically  aggregate  the  utilities  of  each 
individual  within  a  decision  making  group.  The  method  is  based  on  the  assumption  that 
the  utilities  of  each  member  are  independent  (Zahedi,  1996).  The  individual  utilities  are 
combined  by  an  additive  or  multiplicative  function  that  includes  variables  such  as 
weights  and  scaling  constants.  Methods  such  as  the  delegation  process  and  Brock 
method  have  been  created  to  address  the  assignment  of  weights  to  the  utility  functions  of 
the  group  members  (Zahedi,  1996). 

The  result  of  this  method  is  a  group  utility  function  that  addresses  the  question  of 
interest.  This  utility  function  can  be  utilized  to  rank  alternatives  and  select  the  one  that 
maximizes  utility  for  the  group.  This  method  is  mathematically  intense  and  requires  the 
estimation  of  individual  utility  functions  for  all  of  the  group  members.  There  is  no 
requirement  for  the  group  members  to  meet  or  interact  to  use  this  method.  This  method 
does  require  the  use  of  a  facilitator  trained  in  mathematics,  statistics  and  utility  theory. 
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Krzysztofowicz  Group  Consensus  Method. 

This  group  decision  method  is  similar  to  the  group  utility  analysis  by 
mathematically  combining  the  preferences  of  the  individuals  in  the  group.  The  goal  is  a 
group  consensus  function  that  represents  the  consolidated  preference  of  the  group.  It  is 
different  because  individual  utility  functions  are  not  required  and  a  group  consensus 
function  may  not  be  detennined  (Zahedi,  1996). 

This  method  utilizes  subgroups  representing  different  expertise  that  are  utilized  to 
estimate  different  marginal  utility  functions.  These  marginal  utility  functions  represent 
attributes  of  an  overall  group  utility  function.  These  marginal  utility  functions  will  then 
be  combined  by  an  additive  or  multiplicative  function.  The  result  is  a  collection  of 
consensus  points  for  the  group  that  can  be  used  to  create  a  consensus  function.  This 
method  focuses  on  combining  preferences  based  on  attributes  decomposed  from  a 
theoretical  group  utility  function.  This  method  is  mathematically  rigorous  and  requires 
the  use  of  trained  facilitators. 

Zahedi  Group  Consensus  Method. 

The  Zahedi  method  is  similar  to  the  Krzysztofowicz  method  in  that  it  generates 
points  and  a  function  to  represent  the  group  consensus.  This  method  creates  a  consensus 
value  for  all  of  the  alternatives  under  consideration.  “In  the  Zahedi  method,  consensus 
values  and  the  consensus  function  are  obtained  directly  from  the  preference  responses  of 
members.  It  does  not  assume  the  existence  of  utility  axioms  and  does  not  require 
members’  utility  estimation”  (Zahedi,  1996:267). 

Each  group  member  provides  an  interval  score  for  each  alternative  under 
consideration  which  is  used  to  calculate  the  mean  and  standard  deviation.  The  computed 
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statistics  are  then  used  to  determine  the  correlation  between  group  members.  The  next 
step  is  to  create  a  covariance  matrix  for  each  alternative  which  is  then  used  to  calculate 
the  weight  for  each  alternative  for  each  member.  The  final  step  is  to  compute  the 
consensus  point  for  each  alternative  by  combining  the  calculated  mean  and  weight. 

“The  consensus  point  could  be  used  directly  for  selecting  the  alternative  with  the 
highest  consensus  value”  (Zahedi,  1996:267).  A  regression  analysis  utilizing  consensus 
points  and  attributes  will  provide  a  consensus  function  for  the  group  (Zahedi,  1996).  It  is 
clear  that  this  method  is  mathematically  rigorous  and  requires  trained  personnel. 

Weighted  Linear  Combinations. 

The  method  of  weighted  linear  combinations  utilizes  individual  probability 
distributions  aggregated  into  a  group  probability  distribution.  This  method  has  been 
labeled  the  “opinion  pool”  and  is  utilized  for  both  discrete  and  continuous  distributions 
(Seaver,  1978:9).  This  method  attempts  to  identify  the  experts  within  a  group  and 
provide  a  higher  weight  to  their  particular  probability  distributions  when  aggregating  for 
the  group  distribution  (Seaver,  1976). 

There  are  numerous  methods  available  to  detennine  the  weights  that  should  be 
used  to  aggregate  the  individual  probability  distributions.  According  to  Seaver  (1976), 
the  weighting  scheme  utilized  has  no  impact  on  the  quality  of  judgment  produced.  This 
method  assumes  that  the  most  knowledgeable  experts  can  be  identified  and  that  the 
facilitators  are  skilled  in  mathematics. 

Aggregation  Using  Conjugate  Distribution. 

This  method  also  attempts  to  combine  individual  probability  distributions  into  a 
group  probability  distribution.  This  method  assumes  that  the  distributions  obtained  from 
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individuals  are  all  from  the  same  conjugate  family  of  distributions.  “The  group 
probability  distribution  is  determined  by  successive  applications  of  Bayes’  Theorem 
using  all  individually  assessed  distributions”  (Seaver,  1976:33).  The  individual 
probability  distributions  are  weighted  for  aggregation  similar  to  the  weighted  linear 
combination  method. 

The  Expert  Use  Model. 

The  expert  use  model  combines  individual  probability  distributions  through  a 
complex  mathematical  procedure.  According  to  Seaver  (1976),  the  individual 
distributions  are  multiplied  by  a  calibration  function  to  turn  them  into  likelihood 
functions  that  are  combined  using  Bayes’  Theorem.  The  calibration  function  serves  the 
purpose  of  eliminating  bias  and  is  different  based  on  whether  or  not  the  individual 
probability  distributions  are  independent.  This  model  is  only  usable  if  the  individual 
probability  assessments  are  independent  (Seaver,  1976). 

The  Probabilistic  Approach. 

Similar  to  the  expert  use  model,  the  probabilistic  approach  differs  only  in  the  use 
of  conditional  probabilities.  Individual  probability  assessments  and  Bayes’  Theorem  are 
utilized  in  the  same  manner  to  obtain  a  group  probability  distribution.  However,  the 
probabilistic  approach  is  concerned  with  the  probability  of  an  event  occurring  given  the 
subjective  distribution  determined  by  each  individual  in  the  group  (Seaver,  1976).  This  is 
different  from  the  expert  use  model  which  utilizes  the  probability  of  a  subjective 
distribution  given  an  event.  Like  the  expert  use  model,  this  approach  can  only  be  used  if 
the  individual  distributions  are  independent  (Seaver,  1976). 
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Cooperative  Game  Theory. 

This  group  decision  method  concerns  the  cooperation  of  two  or  more  individuals, 
called  players,  in  attempt  to  maximize  their  own  gain.  “In  practice,  the  players  often 
solve  some  optimization  problem  or  consider  some  non-cooperative  game  in  order  to 
arrive  at  the  amount  of  additional  value  available  from  cooperation”  (Lucas,  1996:244). 

Complex  math  is  utilized  to  obtain  solutions  for  cooperative  games.  Three 
different  approaches  that  are  utilized  to  obtain  a  solution  are  the  core,  nucleolus,  and  the 
Shapely  value  (Lucas,  1996).  Cooperative  game  theory,  like  many  of  the  methods 
presented,  requires  the  use  of  involved  mathematics  and  trained  facilitators. 

Chapter  Summary 

In  this  chapter,  the  relationship  between  transformation,  modernization,  and  the 
use  of  MOD  A  for  modernization  planning  has  been  established.  The  first  objective  of 
this  research  effort  has  been  accomplished  through  the  discussion  of  the  CAFPPAT.  The 
role  and  significance  of  experts  and  group  decision  making  is  demonstrated  from  the 
CAFPPAT  review.  Finally,  current  methods  for  obtaining  a  group  decision  from 
collective  experts  were  reviewed.  This  review  of  these  methods  will  facilitate  the 
selection  of  methods  applicable  to  MOD  A  models  such  as  the  CAFPPAT. 
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III.  Methodology 


Chapter  Overview 

The  first  objective  of  this  research  effort,  documentation  of  the  CAFPPAT,  was 
accomplished  in  chapter  II  through  literature  review  and  analysis.  The  second  objective 
of  this  research,  and  the  focus  of  this  chapter,  is  to  identify  the  group  decision  methods 
that  are  applicable  to  MODA.  This  includes  the  group  decision  methods  that  produce 
consensus  inputs  which  can  be  utilized  for  the  structuring,  weighting,  and  scoring 
accomplished  in  MODA  models.  The  result  of  this  effort  will  be  a  taxonomy  table  that 
can  be  used  as  a  reference  for  evaluating  group  decision  methods  for  MODA  models. 

Methodology  Construction 

The  methodology  of  this  study  is  to  map  the  applicable  group  decision  methods  to 
criteria  or  driving  questions  that  would  dictate  their  use  in  MODA.  Chapter  two  provided 
a  list  of  methods  that  will  be  evaluated  for  use  in  MODA.  This  will  be  accomplished 
through  the  review  of  literature  that  documents  the  strengths,  weaknesses,  and  the 
appropriate  uses  of  each  technique.  The  different  decision  contexts  and  settings  that  are 
possible  in  MODA  will  be  incorporated.  The  result  of  this  mapping  effort  will  be  a 
taxonomy  that  will  be  displayed  in  table  format. 

The  resulting  table  is  intended  to  serve  as  a  reference  for  decision  analysts  to 
evaluate  what  group  decision  making  method  would  be  applicable  to  their  particular 
MODA  effort.  The  table  attempts  to  bridge  the  gap  between  usable  group  decision 
methods  and  MODA.  At  a  minimum,  the  resulting  table  will  represent  a  first  effort  at 
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identifying  methods  applicable  to  MODA  which  can  later  be  improved  and  expanded. 
A  visual  representation  of  this  methodology  is  provided  in  Figure  17. 


MANAGEMENT  SCIENCE 


Taxonomy 


Figure  17.  Methodology  Overview  (Greiner,  2002) 

The  table  will  then  be  utilized  to  satisfy  the  third  objective  of  this  research  effort 
concerning  the  CAFPPAT  group  decision  process.  The  observed  CAFPPAT  process  will 
be  compared  to  the  table  in  order  to  conduct  analysis  and  provide  insight.  Ideally,  the 
insight  provided  will  assist  in  improving  the  CAFPPAT  process. 


Group  Decision  Methods  Applicable  to  MODA 

In  order  to  detennine  the  group  decision  methods  that  are  applicable  to  MODA,  a 
few  key  assumptions  have  to  be  made  in  terms  of  the  MODA  environment.  These 
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assumptions  are  based  on  the  observed  CAFPPAT  environment  at  ACC  that  will  be 
described  in  Chapter  4.  These  assumptions  are  limiting  factors,  but  not  restrictive  enough 
to  jeopardize  the  use  of  the  taxonomy  table  in  other  MODA  settings.  These  assumptions 
will  provide  a  baseline  to  evaluate  the  group  decision  methods.  The  assumptions  are  as 
follows: 

1 .  Limited  time  exists  to  complete  the  MODA. 

2.  There  are  a  limited  number  of  analysts/facilitators  available  capable  of 
utilizing  complex  methods. 

3.  Resources  such  as  experts,  facilities,  and  tools  may  be  limited. 

These  conditions  are  restrictive,  but  they  reflect  the  reality  of  many  organizations  that 
would  utilize  MODA.  Organizations  that  do  not  have  these  restrictions  will  still  be  able 
to  utilize  the  information  presented  in  the  taxonomy  table. 

Based  on  the  assumptions  identified  and  literature  reviewed,  four  of  the  group 
decision  methods  presented  in  chapter  two  are  applicable  to  MODA.  These  methods  are 
face  to  face  group  interaction,  voting,  Nominal  Group  Technique,  and  the  Delphi 
Technique.  However,  as  previously  noted,  voting  can  take  on  many  forms  based  on  the 
aggregation  procedures  used.  The  other  three  methods  identified  incorporate  voting  as 
part  of  their  statistical  aggregation  process.  The  type  of  voting  method  to  use  in  face  to 
face  interaction,  Nominal  Group  Technique,  and  the  Delphi  Technique  is  the  subject  of 
separate  research.  Therefore,  voting  will  be  excluded  from  the  taxonomy  created  in  this 
research. 

All  of  the  other  methods  described  in  chapter  two  have  been  excluded  from  the 
taxonomy  table.  These  exclusions  have  been  made  based  on  the  time  requirement, 
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resource  requirement,  or  method  rigor  and  complexity.  Given  a  different  set  of 
assumptions,  these  methods  may  be  applied  to  expand  and  improve  the  taxonomy.  The 
following  sections  review  the  positive  and  negative  aspects,  applicability,  and  appropriate 
use  of  the  three  methods  selected.  One  point  that  is  critical  to  all  three  of  these  group 
decision  methods  is  the  quality  of  the  decision  obtained  is  based  on  the  quality  of  experts 
utilized. 

Face  to  Face  Group  Interaction. 

Face  to  face  group  interaction  is  the  meeting  of  a  group  of  individuals  with  the 
intent  of  reaching  a  consensus  group  answer.  The  major  premise  behind  this  method  of 
group  decision  making  is  that  a  group  of  experts  are  more  likely  to  generate  a  better 
decision  than  a  single  individual.  Consensus  answers  are  obtained  through 
communication  of  ideas  and  opinions  with  compromise  eventually  providing  a  solution. 
According  to  Hornsby  (1994),  “consensus  is  reached  when  all  group  members  accept  the 
final  decision”. 

Face  to  face  interaction  is  flexible  enough  that  it  can  be  applied  to  all  areas  of 
MODA.  Groups  using  this  method  can  structure  models  and  utilize  any  of  the  different 
weighting  schemes  used  in  MODA.  Additionally,  the  ranking  or  scoring  of  alternatives 
can  also  be  accomplished. 

This  method  allows  communication  and  interaction  providing  the  free  flow  of 
information  and  ideas.  Additionally,  it  provides  experts  with  a  sense  of  participation  in 
the  process.  Research  conducted  by  Mahler  (1987:340)  demonstrated  that  face  to  face 
group  members  felt  they  were  able  to  “express  their  views”  more  than  in  structured 
interaction  methods.  Roth  (1995)  found  that  many  experts  “are  not  satisfied  by  a  meeting 
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process  without  face  to  face  interaction”.  Additionally,  Miner  (1979)  discovered  that 
greater  acceptance  of  the  decisions  generated  was  obtained  from  this  method  when 
compared  to  the  nominal  group  technique  and  the  Delphi  technique. 

According  to  Dewar  (1996),  face  to  face  interaction  faces  the  problems  of 
influence  from  dominant  individuals,  irrelevant  conversation,  and  pressure  for 
conformity.  Individuals  can  influence  and  dominate  the  interaction  between  experts 
which,  in  turn,  can  affect  the  consensus  answer  obtained.  Irrelevant  conversation  can 
lead  to  the  formulation  of  a  consensus  decision  taking  longer  than  necessary.  Group 
pressures  to  obtain  a  solution  in  an  expeditious  manner  may  cause  some  ideas  or 
alternatives  to  be  overlooked.  The  use  of  a  neutral  facilitator  that  maintains  the  focus  of 
the  group  and  helps  to  resolve  conflict  improves  the  overall  efficiency  of  the  method. 

The  larger  the  group,  the  more  potential  for  conflict,  disagreement,  and  increased 
time  to  reach  a  consensus  answer.  Although  it  is  possible  to  accomplish  this  method  with 
large  groups  of  experts,  the  appropriate  minimum  number  of  experts  to  reach  a  quality 
consensus  answer  should  be  sought.  Additionally,  it  is  possible  that  a  consensus  answer 
cannot  be  agreed  upon  through  this  method.  This  situation  requires  utilization  of  another 
technique  or  decision  analysis  tool  in  attempt  to  resolve  the  conflict. 

Nominal  Group  Technique. 

The  Nominal  Group  Technique  is  another  group  decision  method  that  is 
applicable  to  all  areas  of  MODA.  This  method  was  created  to  “take  advantage  of  the 
known  superiority  of  group  processes  while  eliminating  the  detrimental  effects”  (Seaver, 
1976:43).  Although  participation  is  not  anonymous,  group  pressures  are  countered 
through  structured  discussion  and  secret  voting. 
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The  benefits  of  utilizing  the  Nominal  Group  Technique  as  a  group  decision 
making  method  are  numerous.  The  method  balances  the  participation  and  influence  of 
the  experts,  produces  more  ideas  than  interactive  groups,  and  results  in  satisfaction  for 
the  group  members  (Dunham,  2002).  Seaver  (1976)  indicates  that  the  level  of 
satisfaction  obtained  from  the  Nominal  Group  Technique  is  greater  than  face  to  face  and 
Delphi  groups  which  were  equivalent.  Additionally,  the  method  reduces  the  need  to 
conform,  de-personalizes  the  issues,  and  provides  a  sense  of  accomplishment  (Dunham, 
2002). 

The  method  is  not  intended  to  take  a  substantial  amount  of  time  to  reach  a  group 
decision.  However,  due  to  iterations  of  idea  generation,  discussion,  and  voting,  there  is  a 
possibility  for  long  group  meetings.  “The  mechanics  of  the  technique  can,  on  occasions, 
become  burdensome  as  it  may  take  considerable  time  to  list  all  of  the  participants’  ideas” 
(Chappie,  1996).  The  issue  of  group  size  also  impacts  that  amount  of  time  it  takes  to 
complete  the  process.  The  more  experts  that  are  used,  the  longer  the  process  will  take. 
This  leads  to  the  questions  of  optimal  group  size  for  the  technique. 

Research  on  the  issue  of  group  size  in  regards  to  the  Nominal  Group  Technique 
provides  mixed  conclusions.  According  to  Fiedler  (1998),  “to  operate  effectively  the 
nominal  group  technique  should  be  small  so  that  each  participant  can  view  the  other”. 
However,  research  conducted  by  the  founders  of  the  method  suggests  that  the  number  of 
experts  does  not  have  to  be  limited  to  a  low  number.  “Nominal  group  processes  can 
accommodate  large  numbers  of  participants  without  the  dysfunctions  of  conventional 
discussion  involving  many  participants”  (Van  de  Yen,  1971).  This  infonnation  leads  to 
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the  conclusion  that  the  technique  will  accommodate  large  numbers,  but  will  be  effective 
and  easier  to  facilitate  with  a  smaller  group. 

The  Nominal  Group  Technique  requires  an  experienced  and  competent  facilitator 
in  order  for  it  to  be  successful  (Fiedler,  1988;  Anderson,  1990).  This  is  critical  because 
research  has  been  done  indicating  that  the  facilitator  can  impact  the  data  collection 
portion  of  the  process.  Chappie  (1996)  discovered  that  group  participants  may  not 
participate  fully  or  be  truthful  based  on  the  actions  of  the  facilitator. 

There  is  some  preparatory  work  that  must  be  accomplished  to  utilize  the 
technique.  Sufficient  facilities  need  to  be  obtained  that  can  seat  the  group  and  the 
necessary  supplies  need  to  be  provided  (Dunham,  2002).  Additionally,  an  opening 
statement  should  be  prepared  to  focus  the  group  on  the  problem  and  elicit  their  full  effort 
(Dunham,  2002).  This  opening  statement  should  educate  the  experts  on  the  problem 
enough  that  they  can  make  sound  decisions. 

The  idea  that  a  decision  obtained  from  the  Nominal  Group  Technique  represents 
group  consensus  has  received  much  debate.  Both  Chappie  (1996)  and  Lomax  (1984) 
suggest  that  results  obtained  from  the  process  may  contradict  the  concept  of  group 
consensus  previously  defined.  However,  the  process  does  produce  a  group  decision  that 
can  be  utilized  for  MODA  input.  This  decision  is  generally  better  than  input  from  one 
individual  because  it  reflects  the  views  of  many  experts. 

Delphi  Technique. 

The  Delphi  Technique  is  the  third  group  decision  method  that  can  be  utilized  for 
MODA.  Like  the  first  two  discussed,  it  is  flexible  enough  for  use  in  all  aspects  of 
MODA.  Delphi  is  an  alternative  to  face  to  face  group  decisions  designed  to  counteract 
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the  negative  effects  of  dominant  personalities,  conversation  not  pursuant  to  the  problem, 
and  open  group  pressure  for  conformity  through  anonymity  (Dewar,  1996).  “The  tenet 
underlying  the  Delphi  technique  is  that  the  consensus  will  improve  with  successive 
rounds  of  anonymous  group  judgments”  (Hornsby,  1994). 

Anonymity  is  one  of  the  main  advantages  of  the  Delphi  Technique.  Anonymity 
serves  the  purpose  of  eliminating  undue  social  pressures  from  individuals  or  environment 
(Rowe,  1999).  Additionally,  it  gives  the  facilitator  the  ability  to  utilize  experts  without 
having  to  gather  them  in  one  place.  This  saves  money  and  time  while  possibly  providing 
an  incentive  for  an  expert  to  participate.  According  to  Macphail  (2001),  Delphi  is 
appropriate  for  groups  that  are  unable  to  meet  face  to  face. 

Large  group  size  is  not  an  issue  that  would  limit  the  use  of  the  Delphi  technique. 
“It  can  be  used  when  the  number  of  participants  exceeds  the  number  with  which  it  is 
impossible  to  conduct  meaningful  face-to-face  discussion”  (Mitchell,  1991:339). 
Alternatively,  small  groups  should  be  avoided  when  using  the  Delphi  method.  Mitchell 
(1991)  points  out  that  the  optimal  size  for  the  group  is  no  less  than  eight  to  ten  members. 

Facilitation  is  not  as  important  an  issue  with  the  Delphi  method  when  compared 
to  the  other  two  identified.  Despite  there  is  no  face  to  face  interaction,  there  is 
communication  of  ideas  and  clarification  between  group  members.  The  interaction  is 
kept  anonymous  and  puts  workload  on  the  process  facilitator  to  ensure  accurate 
communication  of  the  ideas  expressed.  Facilitation  is  required  for  Delphi,  but  it  is  a 
different  form  and  does  not  require  the  people  skills  necessary  for  the  other  two  methods. 

The  overall  amount  of  time  it  takes  to  reach  a  group  decision  using  the  Delphi 
method  can  be  large.  “A  great  deal  of  preparation  is  required  due  to  the  nature  of  written 
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communication”  (Srisoepardani,  1996).  Once  the  process  begins,  the  facilitation  of  this 
method  can  require  a  lot  of  time  to  complete.  This  is  based  on  the  statistics  and  feedback 
that  needs  to  be  consolidated  and  provided  between  rounds.  Additionally,  the  number  of 
rounds  it  takes  to  reach  a  consensus  answer  may  be  lengthy.  Although  four  rounds  is  the 
optimal  amount  in  utilizing  Delphi  (Erffmeyer,  1986),  the  number  can  be  reduced  in  the 
effort  of  reaching  a  decision  faster. 

One  negative  aspect  of  the  Delphi  technique  is  the  lack  of  support  for  the  group 
decision  from  members  that  provided  answers  different  from  the  consensus  (Guzzo, 

1982).  A  separate  negative  aspect  pointed  out  by  Guzzo  (1982),  is  that  the  non-verbal 
interaction  allowed  under  the  method  may  not  necessarily  allow  complete  understanding 
of  issues  involved.  The  Delphi  technique  will  not  work  if  face  to  face  interaction  is 
needed  for  the  benefits  of  “group  spontaneity  and  creative  interaction”  (Souder,  1980). 

The  ability  to  maintain  the  group  of  experts  through  the  entire  process  is  critical  to 
the  Delphi  technique.  According  to  Mitchell  (1991),  high  panel  attrition  is  a  common 
problem  to  Delphi  applications.  “Many  respondents  find  the  exercise  more  burdensome 
than  anticipated.  High  rates  of  attrition  may  mean  that  the  final  results  are  based  upon  an 
unrepresentative  sub  sample  of  the  original  sample”  (Mitchell,  1991:341). 

Another  criticism  is  that  the  Delphi  method  does  not  provide  true  consensus 
(Sackman,  1975),  but  rather  a  statistically  aggregated  consensus  decision  like  the  nominal 
group  technique.  This  may  lead  to  dissatisfaction  for  the  group  members  that  answered 
significantly  different  than  the  consensus  decision.  The  group  member  may  not  accept  or 
support  the  consensus  decision  reached. 
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Taxonomy  Table 

The  review  of  the  three  group  decision  methods  and  their  applicability  to  MODA 
is  captured  in  a  taxonomy  table.  The  applicable  methods  are  displayed  in  the  columns 
and  the  criteria  for  use  are  displayed  in  the  rows.  An  “X”  displayed  at  the  intersection  of 
a  method  and  criteria  indicate  applicability.  Table  3  displays  the  group  decision  methods 
applicable  to  different  aspects  of  MODA. 
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Table  3.  MODA-Group  Decision  Taxonomy 


MODA  -  Criteria/Driving  Questions 


MODA 


Structuring 


Weighting 


Scoring 


Group  Size 


Small  Group  (2-10) 


Medium  Group  (11-30) 


Large  Group  (31+) 


Geographic  Location 


Collocated 


Separated 


Facilitation 


Requires  Experienced  Facilitator/Leadership 


Requires  Facilitator 


Enhanced  Through  Facilitation 


Group  Dynamics 


Open  Discussion  of  Ideas 


Clarification  of  Ideas  -  Structured  Discussion 


Clarification  of  Ideas  -  No  Discussion 


Counters  Dominant  Personalities 


Counters  Irrelevant  Conversation 


Counters  Pressure  to  Conform 


Force  All  Experts  to  Participate 


Possibility  of  Attrition  of  Group  Members 


Expert  Perception 


Expert  Feels  Involved 


Expert  Tends  to  Accept  Decision 


Expert  Feels  Satisfied  With  Process 


Time  Required  to  Complete 


Extensive  Amount  of  Time 


Moderate  Amount  of  Time 


Minimal  Amount  of  Time 


Preparation  Required 


Extensive 


Moderate 


Minimal 


Resources  Required 


Meeting  Place 


Office  Supplies 


Output  From  the  Method 


_ Requires  Complete  Agreement _ 


Determined  Through  Statistical  Aggregation 


Possibility  of  No  Group  Decision 


Possibility  Decision  is  Not  True  Consensus 


Face  to  Face 
Group 
Interaction 


Nominal 

Group 

Technique 
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Chapter  Summary 

The  methodology  utilized  for  this  thesis  effort  has  been  provided  in  this  chapter. 
The  second  objective  of  this  research  effort  has  been  accomplished  through  the 
identification  of  group  decision  making  methods  applicable  to  MODA.  Face  to  face 
interaction,  the  Nominal  Group  Technique,  and  the  Delphi  Technique  were  reviewed  for 
pros  and  cons,  applicability,  and  appropriate  use.  The  result  of  this  review  is  a  taxonomy 
reference  table  to  link  the  decision  methods  to  driving  questions  or  criteria  that  are  of 
concern  when  conducting  MODA.  This  taxonomy  will  be  utilized  to  analyze  the  group 
decision  process  utilized  for  the  CAFPPAT. 
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IV.  CAFPPAT  Group  Decision  Analysis 


Chapter  Overview 

This  chapter  provides  the  documentation  and  analysis  of  the  group  decision 
making  process  utilized  for  CAFPPAT  input.  This  analysis  provides  feedback  and 
constructive  insight  into  the  process  used  by  the  ACC/DRYR  analysis  team. 
Additionally,  the  current  process  is  compared  to  the  taxonomy  table  created  in  Chapter 
III.  This  analysis  identifies  the  characteristics  of  the  CAFPPAT  process  that  support  the 
use  of  the  group  decision  methods  identified  in  the  taxonomy. 

CAFPPAT  Group  Decision  Task 

The  CAFPPAT  group  decision  process  is  utilized  for  the  structuring  and 
weighting  of  the  model  as  well  as  the  scoring  of  solutions.  The  particular  group  decision 
process  observed  for  this  research  effort  focused  on  the  task  of  structuring  and  weighting 
campaign  level  task  dimensions.  This  is  done  to  improve  the  ability  to  evaluate  current 
task  performance  while  eliminating  model  independence  problems  and  time  constraints 
(Hickman,  2002d). 

Two  meetings  were  conducted  to  address  the  neutralization  and  ISR  campaign 
level  tasks  dimensions  respectively.  The  group  decision  process  that  was  observed  for 
these  two  meetings  is  not  unique  to  the  structuring  and  weighting  of  task  dimensions. 

The  same  process  and  methods  are  used  for  all  phases  of  the  CAFPPAT  (Hickman, 
2002d).  Therefore,  the  insight  gained  into  the  group  decision  making  process  is 
applicable  to  the  entire  model. 
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CAFPPAT  Group  Decision  Process 

The  CAFPPAT  group  decision  process  captures  aspects  of  different  group 
decision  methods,  not  strictly  following  any  one  in  particular.  Four  key  aspects  of  group 
decision  making  are  critical  to  analyzing  this  process.  The  setting,  group  composition, 
process  mechanics,  and  facilitation  are  fundamental  to  analyzing  this  group  decision 
method. 

The  setting  of  the  two  group  decision  meetings  was  the  office  of  ACC/DRYR. 

The  office  has  a  central  conference  area  which  is  used  for  face  to  face  group  interaction. 

A  computer,  projector,  and  whiteboard  were  utilized  for  presentation  and  facilitation 
purposes.  Additionally,  poster  sized  model  hierarchies,  charts,  and  task  lists  were 
available  for  the  meetings.  The  setting  allowed  full  participation  of  all  group  members, 
but  the  work  space  available  to  each  individual  was  minimal. 

The  groups  of  SME  were  made  up  of  both  active  duty  military  and  civilian 
employees.  The  group  that  worked  on  the  neutralization  task  dimensions  consisted  of 
nine  individuals  with  eight  of  them  being  active  duty  or  retired  military.  These  nine 
individuals  represented  a  mix  of  operators  from  a  variety  of  aircraft  and  analysts  with 
extensive  experience.  A  few  of  the  SME  utilized  for  this  portion  of  the  CAFPPAT  are 
responsible  for  the  creation  of  modernization  plans.  The  group  that  worked  on  the  1SR 
task  dimensions  consisted  of  five  individuals.  Three  of  the  five  are  intelligence 
professionals  and  are  experienced  in  their  field.  These  groups  contained  military  ranking 
from  captains  to  lieutenant  colonels,  high  ranking  civilians,  and  defense  contractors. 

The  process  that  was  utilized  for  group  decision  making  is  based  upon  face  to  face 
group  interaction.  This  method  is  used  to  discuss  and  debate  the  issues  under 
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consideration.  Individual  knowledge  is  shared  and  different  perspectives  were  presented 
for  group  consideration.  However,  the  consensus  group  decisions  were  obtained  in 
different  ways  throughout  the  two  meetings.  The  difference  was  based  on  the  task 
presented  to  the  group. 

The  structuring  of  task  dimensions  was  completed  using  face  to  face  interaction  to 
revise  a  straw  man,  resulting  in  a  group  consensus  answer.  The  weighting  of  the  task 
dimensions  was  accomplished  by  soliciting  weights  from  each  individual.  These  weights 
were  then  averaged  to  obtain  the  group  weights  for  the  neutralization  task  dimensions. 
The  averaging  served  the  purpose  of  discord  resolution  between  SME.  In  addition,  the 
averaging  saved  time  by  not  requiring  group  agreement  on  the  appropriate  weight.  The 
averages  were  then  presented  to  each  individual  at  a  later  time  for  review  and  feedback. 
Similar  to  the  Delphi  Technique,  numerous  iterations  of  this  review  and  feedback  process 
were  conducted  to  reach  a  consensus  weighting  scheme. 

The  ISR  task  dimension  weights  were  obtained  through  discussion  and 
compromise.  A  possible  reason  for  this  is  the  small  size  of  the  group.  A  summary  of  the 
group  decision  methods  used  for  the  two  meetings  just  discussed  is  presented  in  Table  4. 
Note  that  face  to  face  group  interaction  is  the  primary  method  utilized  with  some 
statistical  aggregation. 


Table  4.  Observed  Grou 

3  Decision  Process  Summary 

Neutralization  Task  Dimensions 

Face  to  Face  Group 
Interaction 

statistical 
Aggregation  - 
Averaging 

Structuring 

X 

Weighting 

X 

ISR  Task  Dimensions 

Structuring 

X 

Weighting 

X 
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Extensive  facilitation  was  provided  by  the  lead  CAFPPAT  analyst  throughout 
both  meetings.  Due  to  the  complexity  of  the  CAFPPAT,  time  was  spent  providing  an 
overview  of  the  model  and  the  specifics  of  the  tasks  that  the  group  needed  to  confront. 
The  lead  analyst  facilitated  the  group  decisions  by  sparking  discussion  and  providing 
interjection  when  necessary.  A  whiteboard,  computer,  and  projection  system  were  used 
by  the  facilitator  throughout  the  meetings.  This  allowed  the  recording  of  preferences  and 
information  to  be  viewed  by  the  entire  group.  Facilitation  was  necessary  to  maintain 
group  focus  and  stop  irrelevant  conversations. 

Analysis  and  Insight 

The  group  decision  process  utilized  for  CAFPPAT  is  infonnal,  flexible,  and 
constrained  by  time  and  resources.  The  process  mixes  aspects  of  face  to  face  interaction 
and  the  Delphi  method.  Different  aggregation  techniques  are  used  to  obtain  group 
consensus  answers  based  on  the  task  performed.  Many  aspects  of  this  group  decision 
process  are  sound  and  contribute  to  obtaining  quality  group  decisions.  However,  there 
are  also  aspects  of  the  process  that  could  be  improved,  resulting  in  better  inputs  into  the 
CAFPPAT. 

Group  Decision  Making  Issues 

There  are  many  issues  facing  the  ACC/D RYR  analysis  team  that  make  utilization 
of  SME  for  CAFPPAT  input  difficult.  Some  of  the  issues  can  be  addressed  by  applying 
the  group  decision  making  methods  identified  in  Chapter  III.  The  remainder  of  the  issues 
will  need  to  be  addressed  through  alternative  means. 
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The  first  issue  is  the  availability  and  utilization  of  experts  for  model  participation. 
The  most  suitable  expert  is  desired  for  each  particular  part  of  the  CAFPPAT  construction. 
Based  on  limitations  such  as  time  constraints  and  mission  requirements,  the  most 
appropriate  SME  is  not  always  available  to  participate  in  the  CAFPPAT  construction. 

The  same  issues  cause  SME  that  have  participated  in  the  CAFPPAT  process  at  one  time 
to  be  unavailable  for  repeated  use.  This  is  an  issue  because  a  replacement  expert  will 
have  no  comprehension  of  the  CAFPPAT  and  will  need  to  be  educated  on  the  model. 

These  issues  of  availability  and  use  of  SME  could  be  countered  by  eliminating  the 
need  to  meet  in  face  to  face  groups.  The  use  of  the  Delphi  method  will  allow  SME  to 
participate  in  the  CAFPPAT  process  without  having  to  gather  in  one  location.  It  will  also 
allow  the  SME  to  work  on  the  CAFPPAT  input  without  investing  the  considerable 
amount  of  time  involved  in  sitting  through  a  group  session.  This  will  allow  the 
participation  of  the  most  appropriate  expert,  as  opposed  to  a  less  experienced,  available 
individual. 

Group  composition  and  group  dynamics  play  a  role  in  the  answers  obtained  from 
the  current  CAFPPAT  group  decision  making  process.  The  observed  group  decision 
process  allows  the  voluntary  exclusion  of  some  members  of  the  group  when  making 
decisions.  This  particular  process  does  not  require  inputs  from  all  of  the  group  members. 
This  allows  some  of  the  group  decisions  to  be  made  by  a  minority  of  the  experts 
participating.  This  lack  of  input  from  some  group  members  could  be  the  result  of  lack  of 
expertise  or  group  social  pressures. 

The  facilitator  made  sure  to  query  for  compliance  on  all  group  decisions  made 
during  the  observed  process.  Every  group  member  was  given  the  opportunity  to  interject 
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and  disagree  with  the  decisions  reached.  However,  the  lack  of  interjection  does  not 
support  the  conclusion  that  a  consensus  answer  was  obtained  each  time. 

The  mixture  of  varying  rank,  experience  levels,  and  personalities  present  during 
the  process  also  suggest  the  existence  of  group  dynamics  capable  of  impacting  the  quality 
of  decision  made.  Individuals  with  strong  personalities  participated  and  often  led  the 
discussion  and  debates.  However,  during  the  observed  process,  there  was  no  blatant 
evidence  of  pressure  for  conformity  or  the  exertion  of  influence.  The  participants  of  the 
observed  process  were  professional  in  dealing  with  each  other.  This  may  not  always  be 
the  case  in  soliciting  group  decisions. 

The  use  of  the  Nominal  Group  Technique  or  the  Delphi  Technique  will  help  to 
prevent  the  negative  effects  that  can  result  from  dominant  personalities  and  pressures  for 
conformity.  These  two  group  decision  methods  will  also  solicit  input  from  every  member 
of  the  group,  ensuring  full  participation.  Both  of  these  techniques  provide  an 
environment  more  conducive  to  a  less  confident  group  member  to  participate. 

One  issue  that  cannot  be  corrected  through  group  decision  methods  is  SME 
comprehension  and  support  of  CAFPPAT.  Due  to  its  high  level  of  complexity  and  depth, 
extensive  effort  is  required  to  educate  SME  on  the  model.  Lack  of  understanding 
translates  into  skepticism  of  the  process  and  the  results  obtained.  Only  through 
communication  and  model  simplifications,  if  possible,  will  this  issue  be  addressed. 

Three  other  issues  surfaced  in  observing  the  current  CAFPPAT  group  decision 
process  that  will  not  be  corrected  through  the  use  of  group  decision  methods.  The 
complexity  of  the  model  often  makes  it  difficult  to  sustain  the  focus  of  the  SME. 
Additionally,  there  sometimes  exists  a  variation  of  definitions  between  SME  when 
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interpreting  model  specifics.  Finally,  the  tasks  of  structuring  and  weighting  were 
accomplished  concurrently  due  to  the  need  to  revise  the  structure.  The  combination  of 
these  three  issues  can  lead  to  conflicts  and  the  stagnation  of  the  group  decision  process. 

Strong  facilitation  may  improve  the  focus  of  SME  on  the  task  at  hand.  The  initial 
facilitation  intended  to  provide  the  necessary  level  of  comprehension  is  critical  to  solving 
this  problem.  Improvement  in  the  ability  to  provide  model  comprehension  will  lead  to 
buy  in  and  improve  the  focus  of  the  SME.  The  conflicts  over  definition  variation  were 
resolved  through  facilitation.  This  is  the  most  appropriate  way  of  dealing  with  this  issue 
based  on  the  fact  that  experts  will  always  have  different  knowledge  and  perspectives  that 
will  need  to  be  reconciled.  Separate  meetings  should  be  conducted  to  structure  and 
weight  the  model.  The  division  of  these  tasks  will  facilitate  efficiency  in  accomplishing 
each  portion  of  the  CAFPPAT.  A  visual  summary  of  the  issues  hindering  CAFPPAT 


group  decision  making  is  provided  in  Figure  18. 


Figure  18.  CAFPPAT  Group  Decision  Issues 
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Taxonomy  Comparative  Analysis 

The  group  decision  making  taxonomy  that  was  presented  in  Chapter  III  provides  a 
basis  for  analyzing  the  CAFPPAT  group  decision  making  environment.  This  analysis 
will  identify  the  method  applicable  for  each  particular  aspect  of  CAFPPAT  group 
decision  making.  The  methods  identified  could  change  based  on  the  dynamic  nature  of 
the  model  and  the  resources  available  for  ACC/DRYR  use. 

Face  to  face  group  interaction,  the  Nominal  Group  Technique,  and  the  Delphi 
Technique  are  all  applicable  for  use  in  the  CAFPPAT.  All  three  of  these  methods  are 
capable  of  structuring,  weighting,  and  scoring.  No  delineation  between  methods  is 
evident  based  on  the  portion  of  the  model  under  consideration. 

Face  to  face  group  interaction  and  the  Nominal  Group  Technique  are  best  suited 
for  the  CAFPPAT  when  considering  the  issue  of  group  size.  There  is  a  high  probability 
that  there  will  be  a  minimal  number  of  experts  available  to  work  on  the  model  at  one 
time.  The  two  methods  identified  are  best  suited  for  small  to  medium  sized  groups.  If 
the  possibility  exists  to  utilize  a  group  size  larger  than  30  experts,  the  Delphi  Technique 
would  be  the  most  applicable  method. 

The  issue  of  geographic  location  is  fundamental  in  choosing  a  group  decision 
method  for  the  CAFPPAT.  If  all  of  the  experts  are  located  in  the  same  location,  any  of 
the  three  methods  can  be  utilized.  However,  if  the  best  experts  to  use  for  the  CAFPPAT 
are  geographically  separated,  the  Delphi  Technique  would  be  the  only  applicable  method. 

Facilitation  is  a  critical  part  of  the  current  CAFPPAT  group  decision  process  due 
to  the  complexity  of  the  model  and  tasks  to  be  accomplished  by  the  groups.  Given  that 
facilitation  is  critical  and  already  exists  in  the  process,  the  Nominal  Group  and  Delphi 
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Techniques  could  be  utilized.  The  use  of  these  two  methods  would  require  the  current 
facilitation  to  be  redirected  to  properly  implement  these  two  methods. 

The  issue  of  group  dynamics  and  the  detrimental  effects  that  they  can  cause  can 
be  addressed  through  the  use  of  the  Nominal  Group  and  Delphi  techniques.  The  use  of 
these  two  methods  eliminates  the  open  discussion  of  ideas,  but  not  clarification  of  the 
ideas  presented.  It  should  be  noted  that  the  use  of  the  Delphi  Technique  for  the 
CAFPPAT  can  result  in  the  attrition  of  group  members  due  to  the  time  it  takes  to 
complete  the  process. 

The  concept  of  expert  perception  and  time  available  to  complete  the  CAFPPAT 
are  two  issues  addressed  by  the  taxonomy.  Face  to  face  group  interaction  is  the  most 
applicable  method  for  the  CAFPPAT  if  the  individual  sense  of  involvement,  satisfaction, 
and  decision  acceptance  of  each  expert  is  critical.  It  should  be  noted  that  the  Delphi 
Technique,  based  on  the  taxonomy,  does  not  satisfy  any  on  these  perception  criteria.  If 
the  time  available  to  complete  applicable  portions  of  the  CAFPPAT  is  minimal,  face  to 
face  group  interaction  and  the  Nominal  Group  Technique  should  be  used.  The  Delphi 
Technique  should  be  utilized  when  an  extensive  amount  of  time  is  available. 

In  congruence  with  the  time  requirement,  the  amount  of  preparation  required  to 
utilize  the  Delphi  Technique  is  extensive.  The  face  to  face  group  interaction  method 
requires  the  least  preparation  to  utilize.  This  method  is  appropriate  when  time  constraints 
and  resources  such  as  facilitators  and  support  personnel  are  not  available.  The  current 
resources  available  to  ACC/DRYR  suggest  the  use  of  this  method  based  on  the 
preparation  criteria. 
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The  final  comparison  between  the  taxonomy  and  the  CAFPPAT  group  decision 
making  process  focuses  on  the  output  required.  If  the  output  from  the  group  decision 
making  method  requires  complete  agreement,  face  to  face  group  interaction  is  the 
appropriate  method.  However,  the  use  of  this  method  allows  the  possibility  of  not 
obtaining  a  group  decision  based  on  discord.  If  the  CAFPPAT  can  use  output  that  is 
determined  through  statistical  aggregation,  the  Nominal  Group  and  Delphi  Techniques 
should  be  utilized.  It  should  be  noted  that  the  results  obtained  from  statistical 
aggregation  should  not  be  considered  true  group  consensus. 

Chapter  Summary 

This  chapter  has  provided  analysis  and  insight  into  the  current  group  decision 
making  process  utilized  for  the  CAFPPAT.  The  current  process  was  identified  based  on 
the  observation  of  two  campaign  level  group  decision  making  meetings.  This  process  is 
evaluated  in  the  context  of  setting,  group  composition,  process  mechanics,  and 
facilitation.  The  issues  hindering  group  decision  making  are  presented  based  on  the 
observations  obtained  from  the  two  meetings.  The  ability  of  the  taxonomy  group 
decision  methods  to  solve  the  issues  identified  is  presented.  Finally,  a  comparison  is 
made  between  the  criteria  provided  in  the  group  decision  making  taxonomy  and  the 
CAFPPAT  group  decision  process.  The  most  applicable  methods  are  identified  for  each 
particular  criteria  of  the  CAFPPAT. 
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V.  Summary  Discussion 


Introduction 

The  intent  of  this  chapter  is  to  provide  an  overview  of  this  research  effort.  The 
motivation  for  this  research  will  be  presented  and  the  objectives  will  be  restated.  A 
summary  of  the  results  will  be  presented  with  the  limitations  of  this  effort  being 
identified.  In  conclusion,  areas  for  further  research  will  be  suggested  for  the  CAFPPAT 
and  group  decision  making. 

Background 

Modernization  planning  is  one  critical  component  of  the  current  Air  Force 
transformation  effort.  The  requirement  to  transform  into  a  more  agile,  expeditionary 
force,  dictates  that  the  Air  Force  strive  for  maximum  return  on  investment.  The 
allocation  of  scarce  taxpayer  dollars  to  sustain  the  current  force  and  build  new  systems  is 
critical  to  the  future  success  of  the  Air  Force.  The  focus  of  modernization  planning  is  the 
capabilities  that  are  required  for  the  Air  Force  to  fight  and  win  in  future  conflicts. 

Capabilities  based  modernization  planning  is  accomplished  through  qualitative 
and  quantitative  methods.  The  use  of  quantitative  decision  analysis  methods  focuses  on 
structuring  complex  decision  problems  and  providing  outputs  useful  to  decision  makers 
in  allocating  scarce  resources.  One  decision  analysis  method  that  is  seeing  an  increase  in 
use  is  multi-objective  decision  analysis  (MODA).  Air  Combat  Command  (ACC)  has 
created  a  MODA  model  to  support  their  capabilities  based  modernization  planning. 

The  Combat  Air  Forces  Planning  and  Programming  Analytical  Tool  analyzes  the 
capabilities  of  the  Combat  Air  Forces  (CAF).  This  model  provides  an  assessment  of  the 
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current  CAF  capabilities  and  identifies  the  areas  where  shortfalls  exist.  This  model 
provides  a  method  to  evaluate  potential  solutions  in  regards  to  improving  or  eliminating 
capability  shortfalls.  Subject  matter  experts  (SME)  are  fundamental  to  the  successful  use 
of  this  type  of  decision  analysis  model.  Experts  are  used  in  lieu  of  the  actual  senior  level 
decision  makers  due  to  time  constraints.  The  use  of  SME  provides  a  broad  range  of 
experience  which  is  then  combined  to  provide  model  inputs. 

The  efficient  utilization  of  experts  directly  relates  to  the  quality  and  fidelity  of  the 
output  generated  from  MOD  A  models.  Numerous  group  decision  methods  exist  that 
allow  a  group  of  people  to  produce  an  output.  However,  differences  in  mathematical 
rigor,  complexity,  and  the  specific  criteria  of  the  MODA  tools  make  the  use  of  some 
group  decision  methods  infeasible.  The  identification  of  group  decision  methods  that  are 
applicable  and  feasible  to  MODA  tools  allows  efficient  use  of  groups  of  experts.  It  is  the 
hope  of  this  research  effort  that  efficient  use  of  experts  will  cause  a  waterfall  effect 
improving  the  output  of  MODA  models  and  modernization  planning. 

Research  Objectives 

This  research  effort  focused  on  satisfying  three  different  goals.  One  goal  was  the 
generalization  and  documentation  of  the  quantitative  decision  making  tool  utilized  by 
ACC  for  modernization  planning.  The  second  goal  was  to  identify  group  decision 
making  methods  that  are  applicable  to  multi-objective  decision  making  models.  Finally, 
the  third  goal  was  to  analyze  and  provide  insight  into  the  group  decision  making  process 
utilized  by  ACC  for  their  decision  tool.  These  three  goals  are  stated  as  the  following 
research  objectives: 
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1. 


Analyze,  generalize  and  document  the  CAFPPAT. 

2.  Based  on  literature,  develop  a  comprehensive  taxonomy  for  group 
decision  making. 

3 .  Observe,  document,  and  analyze  the  group  decision  making  process 
utilized  for  the  CAFPPAT  campaign  level  and  provide  insight. 

Research  Summary 

The  generalization  and  documentation  of  the  CAFPPAT  was  accomplished 
through  literature  review  and  iterations  of  model  familiarization  briefs  by  the 
ACC/DRYR  staff.  The  result  is  an  overview  of  the  model  that  encompasses  the 
motivation,  key  components,  outputs,  and  flexibility  of  use.  The  documentation  provides 
a  general  understanding  of  what  the  model  is  and  how  it  produces  output  in  support  of 
modernization  planning.  This  satisfies  the  first  research  objective  of  providing  a 
documented  reference  of  the  CAFPPAT. 

An  extensive  literature  review  of  decision  theory  identified  three  group  decision 
making  methods  applicable  to  MODA.  The  three  methods  identified  are  appropriate  for 
use  based  on  different  criteria  discovered  in  the  literature  review.  The  methods  and 
criteria  were  combined  into  a  taxonomy  reference  table  satisfying  the  second  research 
objective.  This  table  allows  the  users  of  MODA  to  identify  the  criteria  specific  to  their 
MODA  situation  and  choose  the  most  applicable  group  decision  method. 

The  taxonomy  table  was  then  utilized  to  provide  insight  into  the  current  group 
decision  process  utilized  for  the  CAFPPAT.  The  current  process  was  observed  and 
documented  for  one  portion  of  the  CAFPPAT.  Constructive  insight  was  provided  to 
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identify  the  issues  that  hinder  the  effective  use  of  groups.  Aspects  that  could  be 
improved  through  application  of  the  MODA  applicable  group  decision  methods  were 
identified.  Issues  that  could  not  be  solved  by  the  group  decision  methods  were  also 
identified  and  discussed.  This  part  of  the  research  effort  satisfied  the  third  and  final 
research  objective. 

Recommendations 

The  utilization  of  group  decision  making  methods  is  dependent  upon  the  context 
of  the  MODA  being  conducted.  The  analysis  of  the  CAFPPAT  demonstrated  that  no  one 
method  is  best  suited  for  all  aspects  of  MODA.  There  are  certain  aspects  of  a  MODA 
context  that  force  the  use  of  a  particular  method.  One  example  is  the  geographic 
separation  of  SME  which  dictates  the  use  of  the  Delphi  Technique. 

The  analysts  conducting  MODA  need  to  carefully  evaluate  the  criteria  applicable 
to  their  decision  situation  and  utilize  the  method  best  suited.  A  combination  of  methods 
may  be  utilized  for  different  stages  of  the  MODA  process.  Analysts  need  to  detennine 
which  criteria  are  most  applicable  to  their  decision  context  and  which  are  most  critical  in 
generating  sound  group  outputs.  This  will  allow  the  best  possible  input  into  their  MODA 
models  improving  credibility  and  output. 

Limitations  of  this  Research 

A  limitation  of  this  research  effort  is  the  small  amount  of  group  decision  making 
that  was  actually  observed.  The  documentation  and  analysis  of  the  current  CAFPPAT 
group  decision  making  process  is  based  on  two  meetings.  The  process  observed  at  these 
two  meetings  was  identified  as  universal  to  the  CAFPPAT.  However,  additional  insight 
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and  analysis  could  be  conducted  with  the  observation  of  more  group  meetings.  This 
would  allow  the  generation  of  more  complete  representation  of  the  current  CAFPPAT 
group  decision  process. 

A  second  limitation  is  the  documentation  of  the  CAFPPAT  model.  This  overview 
of  the  CAFPPAT  omits  many  of  the  process  mechanics  critical  to  generating  outputs 
from  the  model.  The  complexity  of  the  model  and  the  focus  on  the  role  of  group  decision 
making  did  not  allow  a  more  detailed  discussion  of  model  mechanics.  A  review 
encompassing  a  higher  level  of  detail  will  provide  a  useful  reference  for  analysts 
constructing  a  model  of  similar  complexity. 

The  final  limitation  of  this  research  deals  with  the  issue  of  group  decision  support 
systems  (GDSS).  GDSS  are  defined  as  “computer-based  systems  and  methods  developed 
to  facilitate  group  decision  making”  (Zahedi,  1996:270).  Improving  communication, 
increasing  participation,  and  providing  a  variety  of  support  functions  to  group  processes 
are  three  examples  of  the  many  uses  of  GDSS  (Zahedi,  1996).  The  use  of  these  systems 
is  increasing  and  the  effect  that  they  may  have  on  the  group  decision  making  taxonomy 
was  not  researched. 

Follow-on  Research 

This  research  effort  provides  many  possibilities  for  further  research  in  regards  to 
both  the  CAFPPAT  and  group  decision  making.  The  CAFPPAT  is  continuously 
evolving  and  improving  to  better  accomplish  its  intended  purpose.  The  ACC/DRYR  staff 
is  pursuing  numerous  efforts  in  an  attempt  to  improve  the  model.  Three  specific  topics 
would  improve  the  fidelity,  credibility,  and  usefulness  of  the  model. 


71 


One  area  of  research  is  the  validation  and  verification  of  the  decision  analysis 
methods  used  for  the  CAFPPAT.  This  research  should  focus  on  the  hierarchies, 
weighting  schemes,  and  the  combination  of  model  components  to  produce  top  level 
outputs.  The  goal  of  this  research  would  be  to  identify  areas  of  deficiency  and  validate 
the  areas  that  do  follow  approved  decision  analysis  methods.  This  would  allow 
ACC/DRYR  to  improve  these  areas  and  increase  the  credibility  of  the  CAFPPAT. 

A  second  topic  useful  to  the  CAFPPAT  is  the  development  of  an  Air  Force 
capabilities  construct.  This  would  require  reviewing  and  updating  the  current  task  lists 
that  represent  Air  Force  capabilities.  The  goal  of  this  effort  would  be  to  ensure  all  the 
necessary  capabilities  were  included.  The  capabilities  construct  would  improve  the  use 
of  the  CAFPPAT  and  could  be  utilized  Air  Force  wide  for  modernization  planning. 

Finally,  the  creation  of  visual  tools  to  display  the  output  generated  from  the 
CAFPPAT  would  increase  the  usefulness  of  the  model  to  senior  level  decision  makers. 
The  current  model  output  is  not  generated  in  a  format  easily  understandable  to  decision 
makers  inexperienced  with  decision  analysis  tools.  This  effort  would  allow  ACC/DRYR 
to  provide  comprehensive  decision  analysis  support  to  ACC  leadership  that  is  simple  to 
use  and  easy  to  understand. 

There  are  numerous  topics  worthy  of  further  research  in  regards  to  group  decision 
making  and  applicability  to  MOD  A  models.  The  verification  and  validation  of  the  group 
decision  making  taxonomy  created  in  this  research  effort  is  one  area  for  future  research. 
This  taxonomy  could  be  evaluated  through  application  on  previously  completed  MODA 
models.  The  results  obtained  through  the  use  of  the  appropriate  group  decision  method 
could  be  compared  to  the  results  originally  obtained.  A  comparative  analysis  of  the 
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results  obtained  may  provide  insight  into  the  relative  worth  of  taxonomy  and  methods 
identified. 

Voting  was  identified  in  Chapter  III  as  a  legitimate  group  decision  method. 
However,  it  was  not  included  in  the  taxonomy  due  to  the  variety  of  forms  of  voting  and 
the  fact  that  the  methods  identified  often  use  some  type  of  voting.  A  topic  of  further 
research  is  to  identify  which  form  of  voting  would  be  most  applicable  for  MODA 
applicable  group  decision  methods.  This  would  allow  analysts  utilizing  group  decision 
methods  to  use  the  form  of  voting  that  will  generate  the  best  group  answer  for  MODA 
input. 

The  group  decision  making  taxonomy  created  in  this  research  could  be  expanded 
to  address  the  use  of  creativity  techniques  that  are  applicable  to  the  structuring  phase  of 
MODA  models.  The  structuring  of  MODA  models  is  critical  in  ensuring  that  a  decision 
analysis  tool  is  comprehensive  in  addressing  the  decision  problem.  The  inclusion  of 
creativity  techniques  and  criteria  for  their  appropriate  use  would  strengthen  the  process  of 
conducting  MODA. 

The  constant  increase  in  technology  and  the  emergence  of  group  decision  support 
systems  (GDSS)  provide  a  new  avenue  to  improve  group  decision  making.  The  impact 
that  GDSS  have  on  the  quality  of  group  decisions  is  worthy  of  research.  Additionally, 
the  identification  of  GDSS  applicable  to  MODA  could  be  used  to  expand  the  taxonomy 
of  group  decision  making  methods. 
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Conclusion 


The  use  of  multi-objective  decision  analysis  is  well  suited  to  the  complex  decision 
problems  faced  both  in  the  public  and  private  sectors.  The  ability  to  provide  structure, 
objectivity,  and  repeatability  when  evaluating  potential  solutions  to  complex  problems 
motivates  the  use  of  MODA.  Subject  matter  experts  are  fundamental  to  the  use  of  this 
form  of  decision  analysis.  The  more  experts  that  can  be  utilized  improve  the  breadth  of 
input  generated. 

The  problem  of  combining  the  input  generated  from  groups  of  experts  is  one 
crucial  to  decision  analysis.  Sound  group  decision  making  methods  are  needed  to  ensure 
the  decision  generated  is  credible  and  useful.  The  relationship  of  these  group  decision 
making  methods  and  MODA  is  critical  in  producing  quality  outputs  useful  to  decision 
makers.  This  relationship  will  only  increase  in  importance  as  the  complexity  of  decisions 
faced  by  leaders  continues  to  increase. 
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Appendix  A.  List  of  Acronyms  and  Abbreviations 


AAR 

Air  to  Air  Refueling 

ACC 

Air  Combat  Command 

ACC/DR 

Directorate  of  Requirements 

ACC/DRPX 

Policy  Analysis 

ACC/DRY 

Analysis  Division 

ACC/DRYR 

Resource  Analysis  Branch 

AFI 

Air  Force  Instruction 

AFPD 

Air  Force  Policy  Directive 

AFPP 

Air  Force  Program  Projection 

AFRAP 

Air  Force  Resource  Allocation  Process 

AFRC 

Air  Force  Reserve  Command 

AFSAA 

Air  Force  Studies  and  Analysis  Agency 

AFSP 

Air  Force  Strategic  Plan 

AFSPC 

Air  Force  Space  Command 

AHP 

Analytical  Hierarchy  Process 

AMC 

Air  Mobility  Command 

ANG 

Air  National  Guard 

ASUS 

Aerospace  Integrated  Investment  Study 

C2 

Command  and  Control 

C4 

Command,  Control,  Communications,  and  Computers 

CAF 

Combat  Air  Forces 

CAFPPAT 

Combat  Air  Forces  Planning  and  Programming  Analytical  Tool 

CONOPS 

Concept  of  Operations 

CSAR 

Combat  Search  and  Rescue 

DoD 

Department  of  Defense 

DPG 

Defense  Planning  Guidance 

EAF 

Expeditionary  Aerospace  Force 

FY 

Fiscal  Year 

GDSS 

Group  Decision  Support  System 

IPP 

Integrated  Planning  Process 

ISR 

Intelligence,  Surveillance,  and  Reconnaissance 

JSF 

Joint  Strike  Fighter 

MAJCOM 

Air  Force  Major  Command 

MAA 

Mission  Area  Assessment 

MAP 

Mission  Area  Plan 

MAT 

Mission  Area  Team 

MAUT 

Multiple  Attribute  Utility  Theory 

MCDM 

Multiple  Criteria  Decision  Making 

MIDB 

Modernized  Integrated  Database 

MNA 

Mission  Needs  Analysis 

MODA 

Multi-Objective  Decision  Analysis 
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MPP 

Modernization  Planning  Process 

MSA 

Mission  Solution  Analysis 

MSP 

Mission  Support  Plan 

PACAF 

Pacific  Air  Forces 

PEM 

Program  Element  Monitor 

POM 

Program  Objective  Memorandum 

PPBS 

Planning,  Programming,  and  Budgeting  System 

RAM 

Resource  Allocation  Model 

SME 

Subject  Matter  Experts 

USAFE 

United  States  Air  Forces  in  Europe 

VFT 

Value  Focused  Thinking 
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Appendix  B.  Classification  of  the  Techniques  and  Methods  Reviewed 


Facilitation/Creativity  Techniques 

Group  Decision  Methods 

Decision  Analysis  Techniques 

A  Questioning  Attitude 

Aggregation  Using  Conjugate  Distributions 

Analytical  Hierarchy  Process 

Affinity  Diagrams 

Cooperative  Game  Theory 

Bayesian  Analysis 

Analogies/Metaphors 

Delphi  Technique 

Conjoint  Measurement 

Attribute  Association 

Expert  Use  Model 

Copeland  Method 

Boundary  Examination 

Face  to  Face  Interaction 

Cost-Benefit  Analysis 

Brain  Writing 

Group  Utility  Analysis 

Decision  Trees 

Brainstorming 

Krzysztofowicz  Group  Consensus  Method 

Fuzzy  Logic 

Bug  List 

Nominal  Group  Technique 

Goal  Programming 

Constructive  Response 

Probabilistic  Approach 

Group  Goal  Programming  Method 

Crawford  Blue  Slip 

Voting 

Group  Naive  Search 

Decomposable  Matrices 

Weighted  Linear  Combinations 

Group  Step  Method 

Devil's  Advocate  Approach 

Zahedi  Group  Consensus  Method 

Hurwicz  Criterion 

Dialectical  Approach 

Influence  Diagrams 

Disjointed  Incrementalism 

Process 

Fish  Bowl 

Matrix  Evaluation 

Fluent  and  Flexible  Thinking 

Maximin  Method 

Force  Field  Analysis 

Multiple  Criteria  Decision  Making 

Go-Around 

Opportunity  Loss  Tables 

Guided  Discussion 

Payoff  Matrices 

Idea  Checklists 

Politometric  Multivariate  Modeling 

Interrogatories  (5Ws/H) 

Principle  of  Insufficient  Reason 

Left/Right  Brain  Alterations 

Probabilistic  Dynamic  Programming 

Lotus  Blossom 

Probability  Models 

Manipulative  Verbs 

Quality  Function  Deployment 

Morphological  Forced  Connections 

Regret  Tables 

Multi-Voting 

Risk  Ranking  Technique 

Peaceful  Setting 

Risk  Reduction  Method 

Please  State  Your  Needs 

Schwartz  Method 

Problem  Reversal 

Utility  Tables 

Progressive  Abstraction 

Utility  Theory 

Put  It  In  the  Hangar 

Q-Methodology 

Take  Five 

The  Gordon  Method 

Why-W hat's  Stopping 

Wildest  Idea 

Wishful  Thinking 

Value  Focused  Thinking 
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